-- Logs begin at Mon 2022-10-24 08:03:16 UTC, end at Mon 2022-10-24 09:13:48 UTC. -- Oct 24 08:36:34.886745 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 24 08:36:34.894279 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: *** Starting uWSGI 2.0.18-debian (64bit) on [Mon Oct 24 08:36:34 2022] *** Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: nodename: dv-cinder-master-3-iscsi Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: machine: x86_64 Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: clock source: unix Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: pcre jit disabled Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: detected number of CPU cores: 4 Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: current working directory: / Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: detected binary path: /usr/bin/uwsgi-core Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: your processes number limit is 47680 Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: your memory page size is 4096 bytes Oct 24 08:36:34.897343 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: detected max file descriptor number: 2048 Oct 24 08:36:34.898056 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: lock engine: pthread robust mutexes Oct 24 08:36:34.898056 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: thunder lock: enabled Oct 24 08:36:34.898056 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 24 08:36:34.898056 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 24 08:36:34.938260 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: Python main interpreter initialized at 0x56370d1969c0 Oct 24 08:36:34.938260 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: python threads support enabled Oct 24 08:36:34.938260 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: your server socket listen backlog is limited to 100 connections Oct 24 08:36:34.938260 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: your mercy for graceful operations on workers is 90 seconds Oct 24 08:36:34.938686 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: mapped 403077 bytes (393 KB) for 2 cores Oct 24 08:36:34.938686 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: *** Operational MODE: preforking *** Oct 24 08:36:34.938968 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: *** uWSGI is running in multiple interpreter mode *** Oct 24 08:36:34.938968 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: spawned uWSGI master process (pid: 93065) Oct 24 08:36:34.939184 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 24 08:36:34.939933 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: spawned uWSGI worker 1 (pid: 93079, cores: 1) Oct 24 08:36:34.941538 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: spawned uWSGI worker 2 (pid: 93080, cores: 1) Oct 24 08:36:34.941538 dv-cinder-master-3-iscsi devstack@keystone.service[93065]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 24 08:36:39.307995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 24 08:36:39.307995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: from cryptography.utils import int_from_bytes Oct 24 08:36:39.308798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 24 08:36:39.308798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: from cryptography.utils import int_from_bytes Oct 24 08:36:39.308921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 24 08:36:39.308921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: from cryptography.utils import int_from_bytes Oct 24 08:36:39.308921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 24 08:36:39.308921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: from cryptography.utils import int_from_bytes Oct 24 08:36:39.360457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 24 08:36:39.360665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 24 08:36:39.360750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 24 08:36:39.360844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] command line args: [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 24 08:36:39.360920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 24 08:36:39.361073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 24 08:36:39.361176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] command line args: [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 24 08:36:39.361251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ================================================================================ {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 24 08:36:39.361329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 24 08:36:39.361488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] admin_token = **** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.361578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ================================================================================ {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 24 08:36:39.361715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] config_dir = [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.361818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] admin_token = **** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.361910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.361994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] config_dir = [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] config_source = [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] debug = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] config_source = [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] debug = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.362977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.363081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] insecure_debug = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.363188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] insecure_debug = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.363293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.363403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.363766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.363870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.363974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] list_limit = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] log_config_append = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] list_limit = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] log_config_append = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364579 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] log_dir = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] log_file = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] log_dir = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.364879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] log_file = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365195 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365825 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.365899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366506 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.366951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.367074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.367163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.367345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.367443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.367637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.367951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] notification_format = cadf {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.368098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] notification_format = cadf {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.368365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.368507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.368864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.39/identity {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.368962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.39/identity {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.369048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] publish_errors = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.369129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] publish_errors = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.369271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.369360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.369499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.369590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.369670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.369743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.370331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.370498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.370590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.370675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.370749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] standard_threads = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.370821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] standard_threads = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.370900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] strict_password_check = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.370973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] strict_password_check = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] use_eventlog = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] use_eventlog = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] use_journal = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] use_journal = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] use_json = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] use_json = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] use_stderr = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] use_stderr = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.371908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] use_syslog = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.372015 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] use_syslog = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.372109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] watch_log_file = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.372264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] watch_log_file = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 24 08:36:39.372345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.372877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.372974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.373050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.373130 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.373203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.373683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.373759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.373842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.373916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.373988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.374467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.374546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.374618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.374699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.374772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.375256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.375333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.375405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.375476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.375556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.375755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.375853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.375939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.376491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.376587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.376679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.376769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.377485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.377622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.377707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.378264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.378359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.378436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.378510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.379201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.379289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.379364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.379939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.380037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.380120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.380780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.380863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.380954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.381037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.381588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.381672 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.381747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.381820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.382564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.382646 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.382720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.382792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.383347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.383430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.383514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.383590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.384262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.384356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.384437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.384981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.385080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.385157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.385237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.385844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.385927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.385999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.386082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.386565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.386649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.386729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.386803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.387009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.387121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.387211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.387680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.387776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.387866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.387942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.388437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.388528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.388610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.388684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.389143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.389242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.389397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.389473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.389940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.390025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.390099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.390227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.390711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.390804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.390878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.390949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.391020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.391532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.391646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.391646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.391832 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.391938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.392482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.392606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.392703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.393168 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.393267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.393341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.393465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.393945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.394045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.394178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.394262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.394782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.394921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.395008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.395489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.395581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.395655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.396188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.396291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.396373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.396447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.396967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.397047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.397121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.397204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.397696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.397782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.397868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.398386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.398481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399785 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.399993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.400504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.400600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.400682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.401164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.401265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.401349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.401423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.401483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.401958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.enabled = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.402045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.enabled = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.402128 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.402204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.402672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.402762 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.402844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.402919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.403001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.403201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.403308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.403393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.403472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.403991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.404085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.404193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.404752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.404839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.404931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.405016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.405539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.405620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.405694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.405767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.406344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.406423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.406506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.406995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.407088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.407152 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.407219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.407293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.411158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.411250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.411790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.411890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.411971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.412058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.412524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.412628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.412707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.412781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.412853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.412925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.413945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] auth.external = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] auth.external = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] auth.mapped = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] auth.mapped = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.414968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] auth.password = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] auth.password = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] auth.token = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415348 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] auth.token = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] catalog.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] catalog.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.415960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] credential.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] credential.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] credential.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] credential.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.416983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.417978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.418786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.419432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.419528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.420025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] federation.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.420110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] federation.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.420225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.420316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.420830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] federation.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.420922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] federation.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.420999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.421073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.421146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.421236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.421448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.421584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.421710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.421834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.421951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422249 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.422954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423848 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.423979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424303 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.424912 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.425927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.426963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.427903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.428975 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.429889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.password = **** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.password = **** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.430946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.431988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.432955 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433081 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.433975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.434893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.435872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.436005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.436121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.436338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.436413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.436520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.436653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.436794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.436919 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437051 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437848 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.437977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.438916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] policy.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] policy.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] receipt.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.439970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] receipt.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440422 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.440977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441051 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] resource.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] resource.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] resource.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] resource.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.441994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.442183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.442327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.442452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] revoke.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.442576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] revoke.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.442700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.442700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.442944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] role.cache_time = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] role.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] role.cache_time = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] role.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] role.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] role.list_limit = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443800 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] role.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.443947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] role.list_limit = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.444980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.445985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.446960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.447966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.448945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449487 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.449915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] token.cache_time = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] token.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] token.cache_time = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.450952 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] token.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] token.provider = fernet {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451218 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] token.provider = fernet {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.451923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] trust.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.452812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] trust.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.453897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.connection = **** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454629 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.454896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.connection = **** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455379 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.455921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.456278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.456353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.456443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.456519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.456607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.456740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.456863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.456987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457374 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.457917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.458040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.458148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.458224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.458307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.458307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=93080) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 24 08:36:39.458443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 24 08:36:39.458443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=93079) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 24 08:36:39.481493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.481702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.481796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.481881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.481960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.482039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.489563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.489716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.520341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.520601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.525387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.525619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.576640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.577161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.627347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.627576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.627814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.628052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.785473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.785660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.785774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.785854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.785957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.786037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.786114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.786189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.786299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.786393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.786508 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.786591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.810355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.810581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.810802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.810902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.811867 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.812084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.823744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.826584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.836986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.837191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.837509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.837741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.837948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.838034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.838167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.838249 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.838384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=93079) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.838469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.838601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.838819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.839033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.839245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=93080) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 24 08:36:39.856394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.856601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 24 08:36:39.856853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.856938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.857063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 24 08:36:39.857164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 24 08:36:39.857400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.857481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.857595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 24 08:36:39.857677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.857868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.857952 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.858128 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.858209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.858647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.858905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.868751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.869035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.869312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.869476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.869601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.869753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.870019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.870297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.877067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.877162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.879983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.880068 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.880291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.880376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.880576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.880685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.880856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.881006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.881137 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.881299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.881379 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.881569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.881651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.881850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.881940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.882126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.882269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.882402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.882541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.882683 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.882807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.882985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.905867 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.906003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.907337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.907454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.907627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.907725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.912311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.912601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.912889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.914112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.915390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.915677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.919827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.920122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.922076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.922364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.925976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.926286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.926502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.926631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.926792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.926941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.927073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.927204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.927335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.927491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.927626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.927800 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.927927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.928169 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.928302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.928442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.944264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.944551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.944844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.945127 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.948451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.949427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.951672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.951975 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.952612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.952909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.956219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.957079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.959396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.960189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.963156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.963455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.963749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.964054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.964368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.964503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.964666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.964805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.964981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.965280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.965572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.965854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.980846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.981151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.981423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.981691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.982115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.982418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.982686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.982946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.988848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.990013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.990972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.991545 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.991932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.992244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.994201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.994485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.995618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.995917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.996226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.996517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.996916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.997294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:39.997582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.997967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.998247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:39.998537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.008036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.009033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.009324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.009612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.010010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.010293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.010588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.010725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.010888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.011276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.011560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.011855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.012175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.012446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.025923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.028044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.028854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.030923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.031976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.033620 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.033784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.035224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.035355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.035535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.035901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.036936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.037229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.037507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.042508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.044069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.044364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.045822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.046958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.047358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.047650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.047937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.048795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.049086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.049370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.049644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.056519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.056816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.057107 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.058333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.058624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.058903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.063958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=93079) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.064278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.064569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.064854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.065132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.065408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.065686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.065991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.066267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.066541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.066676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=93080) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 24 08:36:40.066825 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.066963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.067105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.067247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.067380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.067512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.067644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=93079) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.067805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.067983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.068279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.068573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.068843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.069109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.069376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.069643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.069914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.070221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=93080) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 24 08:36:40.102621 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x56370d1969c0 pid: 93079 (default app) Oct 24 08:36:40.104621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x56370d1969c0 pid: 93080 (default app) Oct 24 08:36:40.106030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23426ec0-bb2a-4622-90f0-d4f4cf321dc5 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:40.106321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23426ec0-bb2a-4622-90f0-d4f4cf321dc5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:40.106601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23426ec0-bb2a-4622-90f0-d4f4cf321dc5 None None] PATH_INFO: `/v3` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:40.110807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1/1] 10.222.0.39 () {50 vars in 791 bytes} [Mon Oct 24 08:36:40 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:36:47.019024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b322e-8c37-49e4-aa2e-d29e24cb6e1c None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:47.019521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b322e-8c37-49e4-aa2e-d29e24cb6e1c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:47.019609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b322e-8c37-49e4-aa2e-d29e24cb6e1c None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:47.023819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1/2] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:47 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:47.029648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:47.030200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:47.030200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:47.101141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: INFO dbcounter [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] Registered counter for database keystone Oct 24 08:36:47.121962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG oslo_db.sqlalchemy.engines [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=93079) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 24 08:36:47.128428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writer thread running {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 24 08:36:47.312033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG passlib.handlers.bcrypt [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=93079) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 24 08:36:47.312435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG passlib.handlers.bcrypt [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=93079) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 24 08:36:47.371669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:47.373322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:47.506072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 2/3] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:47 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 477 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 24 08:36:47.943272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4b322e-8c37-49e4-aa2e-d29e24cb6e1c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:48.013948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: INFO dbcounter [None req-5a4b322e-8c37-49e4-aa2e-d29e24cb6e1c None None] Registered counter for database keystone Oct 24 08:36:48.035658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG oslo_db.sqlalchemy.engines [None req-5a4b322e-8c37-49e4-aa2e-d29e24cb6e1c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=93080) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 24 08:36:48.042933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writer thread running {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 24 08:36:48.162267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4b322e-8c37-49e4-aa2e-d29e24cb6e1c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:48.167176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976735f-5a45-46d4-a780-303cc464ddb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:48.167794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976735f-5a45-46d4-a780-303cc464ddb4 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:48.168075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976735f-5a45-46d4-a780-303cc464ddb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:48.168375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976735f-5a45-46d4-a780-303cc464ddb4 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:48.176607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976735f-5a45-46d4-a780-303cc464ddb4 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:48.227809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING keystone.common.rbac_enforcer.enforcer [None req-5976735f-5a45-46d4-a780-303cc464ddb4 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 24 08:36:48.228081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976735f-5a45-46d4-a780-303cc464ddb4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:48.238183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 2/4] 10.222.0.39 () {58 vars in 1324 bytes} [Mon Oct 24 08:36:47 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 24 08:36:48.245064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:48.262942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5085e69-a71d-4728-8fa6-a8a4a0a80d0c None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:48.267427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc5d65d-6472-45df-908e-4c45473f0743 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:48.268581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc5d65d-6472-45df-908e-4c45473f0743 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:48.268868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc5d65d-6472-45df-908e-4c45473f0743 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:48.269144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc5d65d-6472-45df-908e-4c45473f0743 admin admin] PATH_INFO: `/v3/projects/03b1f5372fb94c0fb009656de2a99622` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:48.270751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc5d65d-6472-45df-908e-4c45473f0743 admin admin] RBAC: Authorizing `identity:get_project(project_id=03b1f5372fb94c0fb009656de2a99622)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:48.323813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING keystone.common.rbac_enforcer.enforcer [None req-9dc5d65d-6472-45df-908e-4c45473f0743 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 24 08:36:48.324185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc5d65d-6472-45df-908e-4c45473f0743 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:48.328798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 3/5] 10.222.0.39 () {56 vars in 1207 bytes} [Mon Oct 24 08:36:48 2022] GET /identity/v3/projects/03b1f5372fb94c0fb009656de2a99622 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:36:50.477533 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa843d-cd2f-4ee5-9aaf-a3d89942d425 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:50.478153 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa843d-cd2f-4ee5-9aaf-a3d89942d425 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:50.478153 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa843d-cd2f-4ee5-9aaf-a3d89942d425 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:50.482116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 3/6] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:50.487876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142bd884-4c53-4619-8a8f-b9ec81c126d1 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:50.488323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142bd884-4c53-4619-8a8f-b9ec81c126d1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:50.488452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142bd884-4c53-4619-8a8f-b9ec81c126d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:50.510558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-142bd884-4c53-4619-8a8f-b9ec81c126d1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:50.512067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-142bd884-4c53-4619-8a8f-b9ec81c126d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:50.519895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 4/7] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:50 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 24 08:36:50.788034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85aa843d-cd2f-4ee5-9aaf-a3d89942d425 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:50.801111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85aa843d-cd2f-4ee5-9aaf-a3d89942d425 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:50.805545 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ae31a3-f9d4-4f11-ac03-3cf936e67e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:50.806160 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae31a3-f9d4-4f11-ac03-3cf936e67e0c admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:50.806442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae31a3-f9d4-4f11-ac03-3cf936e67e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:50.806725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae31a3-f9d4-4f11-ac03-3cf936e67e0c admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:50.814513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ae31a3-f9d4-4f11-ac03-3cf936e67e0c admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:50.816331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ae31a3-f9d4-4f11-ac03-3cf936e67e0c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:50.826204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 4/8] 10.222.0.39 () {58 vars in 1324 bytes} [Mon Oct 24 08:36:50 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:36:50.831889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142bd884-4c53-4619-8a8f-b9ec81c126d1 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:50.841987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142bd884-4c53-4619-8a8f-b9ec81c126d1 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:50.846415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd03765-6935-4575-aa1c-086954ba03e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:50.847043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd03765-6935-4575-aa1c-086954ba03e3 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:50.847327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd03765-6935-4575-aa1c-086954ba03e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:50.847600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd03765-6935-4575-aa1c-086954ba03e3 admin admin] PATH_INFO: `/v3/users/0bb4ebf18f65413bbe4945aec1caead8` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:50.848931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd03765-6935-4575-aa1c-086954ba03e3 admin admin] RBAC: Authorizing `identity:get_user(user_id=0bb4ebf18f65413bbe4945aec1caead8)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:50.850757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd03765-6935-4575-aa1c-086954ba03e3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:50.852418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 5/9] 10.222.0.39 () {56 vars in 1198 bytes} [Mon Oct 24 08:36:50 2022] GET /identity/v3/users/0bb4ebf18f65413bbe4945aec1caead8 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:36:53.095826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742fc1c-3957-412b-aac1-07b357400989 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:53.096395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742fc1c-3957-412b-aac1-07b357400989 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:53.096395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742fc1c-3957-412b-aac1-07b357400989 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:53.097742 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 5/10] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:53.103034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4dab2-a47a-43cb-a73d-d5d6d6710576 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:53.103694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4dab2-a47a-43cb-a73d-d5d6d6710576 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:53.103694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4dab2-a47a-43cb-a73d-d5d6d6710576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:53.122342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-96a4dab2-a47a-43cb-a73d-d5d6d6710576 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:53.123526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-96a4dab2-a47a-43cb-a73d-d5d6d6710576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:53.184730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2205ef-e1c5-4a9b-b534-2172c54b60c0 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:53.185016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2205ef-e1c5-4a9b-b534-2172c54b60c0 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:53.185293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2205ef-e1c5-4a9b-b534-2172c54b60c0 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:53.186627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 6/11] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:36:53.191858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:53.192157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:53.192455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:53.209810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 6/12] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:36:53.234911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG passlib.handlers.bcrypt [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=93080) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 24 08:36:53.235298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG passlib.handlers.bcrypt [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=93080) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 24 08:36:53.285906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:53.287402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:53.388450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 7/13] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:36:53.482027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4dab2-a47a-43cb-a73d-d5d6d6710576 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:53.506891 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:53.590755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4dab2-a47a-43cb-a73d-d5d6d6710576 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:53.627837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b97bd9-9c11-4f19-8ba1-99effb8dfdf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:53.628487 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b97bd9-9c11-4f19-8ba1-99effb8dfdf7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:53.628771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b97bd9-9c11-4f19-8ba1-99effb8dfdf7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:53.629047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b97bd9-9c11-4f19-8ba1-99effb8dfdf7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:53.640428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b97bd9-9c11-4f19-8ba1-99effb8dfdf7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-e2b97bd9-9c11-4f19-8ba1-99effb8dfdf7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:36:53.649722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:36:53.650665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:53.651659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 7/14] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:36:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:36:53.656808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:53.664306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b97bd9-9c11-4f19-8ba1-99effb8dfdf7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:53.751668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:53.763605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72518c59-6c88-4919-bec6-0082b12be3d6 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:53.766009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:53.771150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b97bd9-9c11-4f19-8ba1-99effb8dfdf7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:53.804889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:53.805860 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:53.806199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:53.806471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:53.807341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:36:53.807648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:53.808839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:53.809132 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:53.809456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:53.809741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:53.810017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:53.810712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:53.812091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:53.822822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 8/15] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:36:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:36:53.823207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.sql.core [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 24 08:36:53.823207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 24 08:36:53.823207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [parameters: {'id': '0b774d1d763940e482e07fd1839430d8', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 24 08:36:53.823207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=93079) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 24 08:36:53.828415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:36:53.841192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result.read() Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 24 08:36:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application yield resource Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.session.rollback() Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.force_reraise() Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise self.value Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application session.commit() Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 24 08:36:53.844922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._prepare_impl() Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.session.flush() Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._flush(objects) Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application compat.raise_( Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application flush_context.execute() Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rec.execute(self) Oct 24 08:36:53.846933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 24 08:36:53.849121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result.read() Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:36:53.851163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [parameters: {'id': '0b774d1d763940e482e07fd1839430d8', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 24 08:36:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:36:53.855395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 8/16] 10.222.0.39 () {60 vars in 1137 bytes} [Mon Oct 24 08:36:53 2022] POST /identity/v3/domains => generated 190 bytes in 179 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:36:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:53.942208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c05c8ef-1472-4952-9ce4-dc2cc5a9bf5a None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:53.962267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f5dec-b06b-4521-8720-60d4fd0b1789 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:53.977767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6ee010-4703-41e9-8023-b729772bd05c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:53.978636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6ee010-4703-41e9-8023-b729772bd05c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:53.979010 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6ee010-4703-41e9-8023-b729772bd05c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:53.979300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6ee010-4703-41e9-8023-b729772bd05c None admin] PATH_INFO: `/v3/users/0bb4ebf18f65413bbe4945aec1caead8` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:53.980874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6ee010-4703-41e9-8023-b729772bd05c None admin] RBAC: Authorizing `identity:get_user(user_id=0bb4ebf18f65413bbe4945aec1caead8)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:53.982351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6ee010-4703-41e9-8023-b729772bd05c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:53.983964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 9/17] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:36:53 2022] GET /identity/v3/users/0bb4ebf18f65413bbe4945aec1caead8 => generated 247 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:36:53.989959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6ee010-4703-41e9-8023-b729772bd05c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:54.006204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:54.006853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:54.007187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:54.007460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:54.018604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:54.020111 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:54.027095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:54.028396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:36:54.037788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:36:54.038717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:36:54.039560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:36:54.039560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:36:54.039560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:54.039560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 9/18] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:36:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:36:54.046806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:54.102792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6ee010-4703-41e9-8023-b729772bd05c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:54.139921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5948af40-1de4-405c-a2d9-5f3a0ab50cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:54.140602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948af40-1de4-405c-a2d9-5f3a0ab50cd4 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:54.140881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948af40-1de4-405c-a2d9-5f3a0ab50cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:54.141149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948af40-1de4-405c-a2d9-5f3a0ab50cd4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:54.142277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5948af40-1de4-405c-a2d9-5f3a0ab50cd4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:54.143724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5948af40-1de4-405c-a2d9-5f3a0ab50cd4 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:54.145555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 10/19] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:36:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:36:54.151349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5948af40-1de4-405c-a2d9-5f3a0ab50cd4 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:54.152685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005899e7-38eb-4637-ac18-45ca53d3fc2c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:54.187422 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e97c472-85fa-4fad-93c5-c1b6df19f01a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:54.188025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e97c472-85fa-4fad-93c5-c1b6df19f01a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:54.188335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e97c472-85fa-4fad-93c5-c1b6df19f01a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:54.188607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e97c472-85fa-4fad-93c5-c1b6df19f01a None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:54.189243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e97c472-85fa-4fad-93c5-c1b6df19f01a None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:36:54.189550 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e97c472-85fa-4fad-93c5-c1b6df19f01a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:54.190923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e97c472-85fa-4fad-93c5-c1b6df19f01a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:54.209535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 10/20] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:36:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:36:54.263543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5948af40-1de4-405c-a2d9-5f3a0ab50cd4 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:54.265333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:54.303771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c1358b-5795-4c1e-865b-262e3ae15dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:54.304489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c1358b-5795-4c1e-865b-262e3ae15dd4 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:54.304766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c1358b-5795-4c1e-865b-262e3ae15dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:54.305042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c1358b-5795-4c1e-865b-262e3ae15dd4 None admin] PATH_INFO: `/v3/domains/default/users/0bb4ebf18f65413bbe4945aec1caead8/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:54.307087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c1358b-5795-4c1e-865b-262e3ae15dd4 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=0bb4ebf18f65413bbe4945aec1caead8, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:54.308642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c1358b-5795-4c1e-865b-262e3ae15dd4 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:54.427052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 11/21] 10.222.0.39 () {58 vars in 1361 bytes} [Mon Oct 24 08:36:54 2022] PUT /identity/v3/domains/default/users/0bb4ebf18f65413bbe4945aec1caead8/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:36:56.708986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e136a2-a1f0-4251-bf0a-e5901211458e None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:56.709522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e136a2-a1f0-4251-bf0a-e5901211458e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:56.709522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e136a2-a1f0-4251-bf0a-e5901211458e None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:56.710861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 11/22] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:56.726743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34281b6d-d6f7-4883-a6b1-edbd2550a4b6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:56.727220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34281b6d-d6f7-4883-a6b1-edbd2550a4b6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:56.727724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34281b6d-d6f7-4883-a6b1-edbd2550a4b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:56.761701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-34281b6d-d6f7-4883-a6b1-edbd2550a4b6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:56.763461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-34281b6d-d6f7-4883-a6b1-edbd2550a4b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:56.859060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 12/23] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:36:57.243922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed36a3f-c153-4f1b-8259-08f6bd75cdbc None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:57.243922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed36a3f-c153-4f1b-8259-08f6bd75cdbc None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:57.243922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed36a3f-c153-4f1b-8259-08f6bd75cdbc None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:57.245051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 12/24] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:57.258136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bf4529-d22d-4a6a-8edd-91bb6dfb482e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:57.258793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bf4529-d22d-4a6a-8edd-91bb6dfb482e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:57.259857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bf4529-d22d-4a6a-8edd-91bb6dfb482e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:57.290273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-80bf4529-d22d-4a6a-8edd-91bb6dfb482e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:57.292221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-80bf4529-d22d-4a6a-8edd-91bb6dfb482e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:57.317887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8a80ed-e467-4a9f-9b4d-c1a54a49a447 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:57.318529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8a80ed-e467-4a9f-9b4d-c1a54a49a447 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:57.318993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8a80ed-e467-4a9f-9b4d-c1a54a49a447 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:57.320547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 13/25] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:57.326102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540bed4-c6de-4ff7-8e1c-4c3011ce1e33 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:57.326663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540bed4-c6de-4ff7-8e1c-4c3011ce1e33 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:57.327247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540bed4-c6de-4ff7-8e1c-4c3011ce1e33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:57.364170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-1540bed4-c6de-4ff7-8e1c-4c3011ce1e33 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:57.366003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1540bed4-c6de-4ff7-8e1c-4c3011ce1e33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:57.446611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 13/26] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:36:57.448047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bf4529-d22d-4a6a-8edd-91bb6dfb482e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:57.507140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:57.521099 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:57.541220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 14/27] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:36:57.710502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bf4529-d22d-4a6a-8edd-91bb6dfb482e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:57.783648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:57.784623 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:57.785120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:57.785620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:57.820696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:57.824640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:36:57.849148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:36:57.851054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:36:57.853480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:36:57.853480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:36:57.853480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:36:57.853480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 14/28] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:36:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 403 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:36:57.861530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1540bed4-c6de-4ff7-8e1c-4c3011ce1e33 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:57.925103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:58.083698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1540bed4-c6de-4ff7-8e1c-4c3011ce1e33 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:58.128682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:58.151344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:58.152556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.153838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.154524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.156110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:36:58.156446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:58.157878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:58.171516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 15/29] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:36:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:36:58.172580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:58.178317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3586e91-46f5-4751-8f7a-970987c3f7fc None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:58.223522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fabea6-8d39-4adf-9034-4af631833ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:58.224589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fabea6-8d39-4adf-9034-4af631833ccf None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.225144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fabea6-8d39-4adf-9034-4af631833ccf None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.225667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fabea6-8d39-4adf-9034-4af631833ccf None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.226674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fabea6-8d39-4adf-9034-4af631833ccf None admin] RBAC: Authorizing `identity:create_role()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:58.228534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fabea6-8d39-4adf-9034-4af631833ccf None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:58.234483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:58.258725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 15/30] 10.222.0.39 () {60 vars in 1131 bytes} [Mon Oct 24 08:36:57 2022] POST /identity/v3/roles => generated 219 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:36:58.260796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f28e1-f5fe-4806-8fae-ba43825c4f38 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.261304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f28e1-f5fe-4806-8fae-ba43825c4f38 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.261819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f28e1-f5fe-4806-8fae-ba43825c4f38 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.263474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 16/31] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:58.269211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80f28e1-f5fe-4806-8fae-ba43825c4f38 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:58.347055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3168767e-4d60-4a6f-828d-a22638065475 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:58.403642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80f28e1-f5fe-4806-8fae-ba43825c4f38 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:58.428229 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d10d29-78c6-4808-8390-da141ffb2ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:58.428889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d10d29-78c6-4808-8390-da141ffb2ad6 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.429529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d10d29-78c6-4808-8390-da141ffb2ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.429809 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d10d29-78c6-4808-8390-da141ffb2ad6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.446719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d10d29-78c6-4808-8390-da141ffb2ad6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-f4d10d29-78c6-4808-8390-da141ffb2ad6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:36:58.459102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:36:58.460065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:36:58.460881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 16/32] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:36:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:36:58.467802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f41e5c-6345-4355-9ab9-98b6e8b12ec2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.468124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f41e5c-6345-4355-9ab9-98b6e8b12ec2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.469535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f41e5c-6345-4355-9ab9-98b6e8b12ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.491982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b72f3c-0350-49b2-be72-a1fc5785698c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:58.492715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b72f3c-0350-49b2-be72-a1fc5785698c None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.493053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b72f3c-0350-49b2-be72-a1fc5785698c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.495984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b72f3c-0350-49b2-be72-a1fc5785698c None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.496726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b72f3c-0350-49b2-be72-a1fc5785698c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:58.498281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b72f3c-0350-49b2-be72-a1fc5785698c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:58.504771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:58.515075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a8f41e5c-6345-4355-9ab9-98b6e8b12ec2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:58.516345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a8f41e5c-6345-4355-9ab9-98b6e8b12ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:58.540669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 17/33] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:36:58 2022] POST /identity/v3/projects => generated 301 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:36:58.541680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b72f3c-0350-49b2-be72-a1fc5785698c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:58.655656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 17/34] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:36:58.657341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d67823-9716-452f-84b4-ad9558c6f646 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.657837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d67823-9716-452f-84b4-ad9558c6f646 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.658336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d67823-9716-452f-84b4-ad9558c6f646 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.665368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 18/35] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:58.667431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8664137-d5c6-48db-8d7b-2e1c4f34bcaa None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.667431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8664137-d5c6-48db-8d7b-2e1c4f34bcaa None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.667431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8664137-d5c6-48db-8d7b-2e1c4f34bcaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.678750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b72f3c-0350-49b2-be72-a1fc5785698c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:58.714166 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a8664137-d5c6-48db-8d7b-2e1c4f34bcaa None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:58.715412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a8664137-d5c6-48db-8d7b-2e1c4f34bcaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:58.720915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:58.721607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.721894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.722169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.722870 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:36:58.723232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:58.724708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:58.736651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 18/36] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:36:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:36:58.745213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:58.780435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:58.810548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:58.833047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 19/37] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:36:58.834971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087db970-d0c0-4d2d-9e6c-135cbc87d049 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.835256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087db970-d0c0-4d2d-9e6c-135cbc87d049 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.835530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087db970-d0c0-4d2d-9e6c-135cbc87d049 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.838304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 20/38] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:36:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:36:58.845604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d6bde-4db7-4c16-88d3-4fa0d55d5368 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.846090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d6bde-4db7-4c16-88d3-4fa0d55d5368 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.846533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d6bde-4db7-4c16-88d3-4fa0d55d5368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.869806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2b6d6bde-4db7-4c16-88d3-4fa0d55d5368 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:36:58.871810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2b6d6bde-4db7-4c16-88d3-4fa0d55d5368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:36:58.876367 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dd5156-f3be-4426-b700-ca8a4e29c017 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:58.912991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3435f121-994a-4112-8825-81c389e80290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:58.913304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3435f121-994a-4112-8825-81c389e80290 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:58.913721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3435f121-994a-4112-8825-81c389e80290 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:58.914003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3435f121-994a-4112-8825-81c389e80290 None admin] PATH_INFO: `/v3/users/0bb4ebf18f65413bbe4945aec1caead8` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:58.915223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3435f121-994a-4112-8825-81c389e80290 None admin] RBAC: Authorizing `identity:get_user(user_id=0bb4ebf18f65413bbe4945aec1caead8)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:58.916717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3435f121-994a-4112-8825-81c389e80290 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:58.918484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 19/39] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:36:58 2022] GET /identity/v3/users/0bb4ebf18f65413bbe4945aec1caead8 => generated 247 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:36:58.924554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3435f121-994a-4112-8825-81c389e80290 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:58.951983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 21/40] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:36:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:36:59.010055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6d6bde-4db7-4c16-88d3-4fa0d55d5368 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:59.046729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3435f121-994a-4112-8825-81c389e80290 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:59.063509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:59.082963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2f5c91-3b7a-45dc-b4ea-46bfe79cdbce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:59.083717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2f5c91-3b7a-45dc-b4ea-46bfe79cdbce None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:59.084071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2f5c91-3b7a-45dc-b4ea-46bfe79cdbce None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:59.084453 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2f5c91-3b7a-45dc-b4ea-46bfe79cdbce None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:59.085654 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2f5c91-3b7a-45dc-b4ea-46bfe79cdbce None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:59.087242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2f5c91-3b7a-45dc-b4ea-46bfe79cdbce None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:59.089176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 20/41] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:36:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:36:59.094828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2f5c91-3b7a-45dc-b4ea-46bfe79cdbce None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:59.137786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:59.142019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6d6bde-4db7-4c16-88d3-4fa0d55d5368 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:59.182819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1b9b2e-2257-4976-b9e5-e24265780032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:59.183320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1b9b2e-2257-4976-b9e5-e24265780032 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:59.183604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1b9b2e-2257-4976-b9e5-e24265780032 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:59.183875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1b9b2e-2257-4976-b9e5-e24265780032 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:59.185028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1b9b2e-2257-4976-b9e5-e24265780032 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:59.186769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1b9b2e-2257-4976-b9e5-e24265780032 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:59.211164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2f5c91-3b7a-45dc-b4ea-46bfe79cdbce None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:59.249385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:59.250076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:59.250355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:59.250821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:59.251740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.domain.id=default, user.id=0bb4ebf18f65413bbe4945aec1caead8) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:36:59.252053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:59.253500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:59.262461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 21/42] 10.222.0.39 () {56 vars in 1322 bytes} [Mon Oct 24 08:36:59 2022] GET /identity/v3/role_assignments?user.id=0bb4ebf18f65413bbe4945aec1caead8&scope.domain.id=default&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 530 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:36:59.263358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:59.319852 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:59.327797 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 22/43] 10.222.0.39 () {60 vars in 1131 bytes} [Mon Oct 24 08:36:59 2022] POST /identity/v3/roles => generated 225 bytes in 318 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:36:59.327797 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1b9b2e-2257-4976-b9e5-e24265780032 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:59.412733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9275cb-b682-4a87-896d-31e27b0c5a55 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:59.458896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1b9b2e-2257-4976-b9e5-e24265780032 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:59.465721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb34c0-61d6-4d80-888c-0f0abf410783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:59.466379 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb34c0-61d6-4d80-888c-0f0abf410783 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:59.466665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb34c0-61d6-4d80-888c-0f0abf410783 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:59.466940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb34c0-61d6-4d80-888c-0f0abf410783 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:59.468114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deb34c0-61d6-4d80-888c-0f0abf410783 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:59.469616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deb34c0-61d6-4d80-888c-0f0abf410783 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:59.471260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 22/44] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:36:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:36:59.477154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb34c0-61d6-4d80-888c-0f0abf410783 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:36:59.504046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57d8a22-5725-40d7-9bae-08e6ec5704eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:59.504724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d8a22-5725-40d7-9bae-08e6ec5704eb None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:59.505294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d8a22-5725-40d7-9bae-08e6ec5704eb None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:59.505577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d8a22-5725-40d7-9bae-08e6ec5704eb None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:59.506254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d8a22-5725-40d7-9bae-08e6ec5704eb None admin] RBAC: Authorizing `identity:create_role()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:59.509037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d8a22-5725-40d7-9bae-08e6ec5704eb None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:59.533434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 23/45] 10.222.0.39 () {60 vars in 1131 bytes} [Mon Oct 24 08:36:59 2022] POST /identity/v3/roles => generated 223 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:36:59.607239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb34c0-61d6-4d80-888c-0f0abf410783 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:36:59.627470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:36:59.647256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62686c4-eeb5-4321-b226-b1d1b8f8eb3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:36:59.647881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62686c4-eeb5-4321-b226-b1d1b8f8eb3f None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:36:59.648187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62686c4-eeb5-4321-b226-b1d1b8f8eb3f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:36:59.648461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62686c4-eeb5-4321-b226-b1d1b8f8eb3f None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:36:59.649134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62686c4-eeb5-4321-b226-b1d1b8f8eb3f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:36:59.651125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62686c4-eeb5-4321-b226-b1d1b8f8eb3f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:36:59.679930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 23/46] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:36:59 2022] POST /identity/v3/projects => generated 312 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:37:01.862463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2dd2d2-e118-4c71-a1b4-782680887f12 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:01.862984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2dd2d2-e118-4c71-a1b4-782680887f12 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:01.862984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2dd2d2-e118-4c71-a1b4-782680887f12 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:01.864378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 24/47] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:01.870441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae98cc-00d5-41a6-8f08-325a39d510ff None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:01.870833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae98cc-00d5-41a6-8f08-325a39d510ff None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:01.870996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae98cc-00d5-41a6-8f08-325a39d510ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:01.888777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-9eae98cc-00d5-41a6-8f08-325a39d510ff None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:01.889969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9eae98cc-00d5-41a6-8f08-325a39d510ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:01.955801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 24/48] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:02.224356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2dd2d2-e118-4c71-a1b4-782680887f12 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:02.246651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:02.435717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2dd2d2-e118-4c71-a1b4-782680887f12 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:02.468862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6a1d82-f9d2-4f18-9024-a4057084af48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:02.469483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a1d82-f9d2-4f18-9024-a4057084af48 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:02.469767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a1d82-f9d2-4f18-9024-a4057084af48 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:02.470044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a1d82-f9d2-4f18-9024-a4057084af48 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:02.471239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a1d82-f9d2-4f18-9024-a4057084af48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:02.472716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a1d82-f9d2-4f18-9024-a4057084af48 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:02.474265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 25/49] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:02.480302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eae98cc-00d5-41a6-8f08-325a39d510ff None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:02.573811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:02.577342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eae98cc-00d5-41a6-8f08-325a39d510ff None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:02.612252 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b38b1b-7656-4a82-8ba5-50333bc5a568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:02.613366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b38b1b-7656-4a82-8ba5-50333bc5a568 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:02.613640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b38b1b-7656-4a82-8ba5-50333bc5a568 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:02.613904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b38b1b-7656-4a82-8ba5-50333bc5a568 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:02.614544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b38b1b-7656-4a82-8ba5-50333bc5a568 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:02.616253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b38b1b-7656-4a82-8ba5-50333bc5a568 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:02.641469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 25/50] 10.222.0.39 () {60 vars in 1140 bytes} [Mon Oct 24 08:37:02 2022] POST /identity/v3/projects => generated 298 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:04.831145 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5509201-08c9-4a4c-b3ed-ecb30fb243ce None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:04.831703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5509201-08c9-4a4c-b3ed-ecb30fb243ce None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:04.831703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5509201-08c9-4a4c-b3ed-ecb30fb243ce None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:04.833039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 26/51] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:04.838769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71daa00-d9b1-40cf-be26-3b542f963331 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:04.839150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71daa00-d9b1-40cf-be26-3b542f963331 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:04.839321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71daa00-d9b1-40cf-be26-3b542f963331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:04.858413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c71daa00-d9b1-40cf-be26-3b542f963331 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:04.859604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c71daa00-d9b1-40cf-be26-3b542f963331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:05.027509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 26/52] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:05.298034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5509201-08c9-4a4c-b3ed-ecb30fb243ce None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:05.391001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:05.406801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5509201-08c9-4a4c-b3ed-ecb30fb243ce None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:05.440595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb31b71-c736-4623-b926-96a614af6ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:05.441215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb31b71-c736-4623-b926-96a614af6ade None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:05.441496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb31b71-c736-4623-b926-96a614af6ade None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:05.441768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb31b71-c736-4623-b926-96a614af6ade None admin] PATH_INFO: `/v3/domains/default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:05.442870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb31b71-c736-4623-b926-96a614af6ade None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:05.444326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb31b71-c736-4623-b926-96a614af6ade None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:05.445878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 27/53] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:05.451582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71daa00-d9b1-40cf-be26-3b542f963331 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:05.530237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:05.550366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71daa00-d9b1-40cf-be26-3b542f963331 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:05.583433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878c1505-fc07-4097-8cd0-8971592ac3f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:05.584038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878c1505-fc07-4097-8cd0-8971592ac3f2 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:05.584345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878c1505-fc07-4097-8cd0-8971592ac3f2 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:05.584622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878c1505-fc07-4097-8cd0-8971592ac3f2 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:05.585276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878c1505-fc07-4097-8cd0-8971592ac3f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:05.586703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878c1505-fc07-4097-8cd0-8971592ac3f2 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:05.613559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 27/54] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:05 2022] POST /identity/v3/users => generated 275 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:37:08.983091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24ad22-d8bd-4ad4-a1d6-8bd8af041e70 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:08.984093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24ad22-d8bd-4ad4-a1d6-8bd8af041e70 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:08.984617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24ad22-d8bd-4ad4-a1d6-8bd8af041e70 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:08.986196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 28/55] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:09.004656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e409c4a4-2348-45ed-b323-15cd4196a7be None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.005116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e409c4a4-2348-45ed-b323-15cd4196a7be None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.005189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e409c4a4-2348-45ed-b323-15cd4196a7be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.028453 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e409c4a4-2348-45ed-b323-15cd4196a7be None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:09.029745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e409c4a4-2348-45ed-b323-15cd4196a7be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:09.122432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 28/56] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:09.185744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806feb06-81b6-472c-a334-e070e2163daf None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.186058 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806feb06-81b6-472c-a334-e070e2163daf None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.186321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806feb06-81b6-472c-a334-e070e2163daf None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.187935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 29/57] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:09.193174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef55b8a5-4188-4b0a-86ff-9e6d77c3e0bb None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.193466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef55b8a5-4188-4b0a-86ff-9e6d77c3e0bb None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.193749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef55b8a5-4188-4b0a-86ff-9e6d77c3e0bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.216445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ef55b8a5-4188-4b0a-86ff-9e6d77c3e0bb None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:09.217685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ef55b8a5-4188-4b0a-86ff-9e6d77c3e0bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:09.288837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:09.303964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 29/58] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:09.315211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beed855-ae76-44c1-90f1-5a57b67949ef None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.315736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beed855-ae76-44c1-90f1-5a57b67949ef None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.316197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beed855-ae76-44c1-90f1-5a57b67949ef None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.318113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 30/59] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:09.324418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737acbac-3a45-47cd-b426-5b16e297157a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.324712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737acbac-3a45-47cd-b426-5b16e297157a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.324991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737acbac-3a45-47cd-b426-5b16e297157a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.349507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-737acbac-3a45-47cd-b426-5b16e297157a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:09.351190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-737acbac-3a45-47cd-b426-5b16e297157a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:09.446968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 30/60] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:09.461243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8beed855-ae76-44c1-90f1-5a57b67949ef None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:09.482670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5fb3e5-f9b8-4706-b1a9-9082f65ed811 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.483205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5fb3e5-f9b8-4706-b1a9-9082f65ed811 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.483627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5fb3e5-f9b8-4706-b1a9-9082f65ed811 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.485339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 31/61] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:37:09.492591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc8ff6-f8b4-4363-bd84-22d9b75a2150 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.493272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc8ff6-f8b4-4363-bd84-22d9b75a2150 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.494346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc8ff6-f8b4-4363-bd84-22d9b75a2150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.516448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-18fc8ff6-f8b4-4363-bd84-22d9b75a2150 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:09.518274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-18fc8ff6-f8b4-4363-bd84-22d9b75a2150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:09.599940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 32/62] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:09.652951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fc8ff6-f8b4-4363-bd84-22d9b75a2150 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:09.686174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8beed855-ae76-44c1-90f1-5a57b67949ef None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:09.744835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:09.773392 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f09d76-5a6c-48a6-98b5-5e907e2a59e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:09.774041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f09d76-5a6c-48a6-98b5-5e907e2a59e9 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.774324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f09d76-5a6c-48a6-98b5-5e907e2a59e9 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.774665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f09d76-5a6c-48a6-98b5-5e907e2a59e9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.775576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fc8ff6-f8b4-4363-bd84-22d9b75a2150 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:09.800712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f09d76-5a6c-48a6-98b5-5e907e2a59e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-15f09d76-5a6c-48a6-98b5-5e907e2a59e9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:09.812370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:09.813474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:09.814343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 31/63] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:09.818669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f09d76-5a6c-48a6-98b5-5e907e2a59e9 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:09.818986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74334264-3fe7-4a59-8e07-3ca8d180c5fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:09.820098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74334264-3fe7-4a59-8e07-3ca8d180c5fa None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:09.820552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74334264-3fe7-4a59-8e07-3ca8d180c5fa None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:09.820964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74334264-3fe7-4a59-8e07-3ca8d180c5fa None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:09.840544 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:09.842132 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74334264-3fe7-4a59-8e07-3ca8d180c5fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-74334264-3fe7-4a59-8e07-3ca8d180c5fa None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:09.852371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:09.854026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:09.854981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:09.857391 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 33/64] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:37:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:37:09.858978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74334264-3fe7-4a59-8e07-3ca8d180c5fa None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:09.980317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f09d76-5a6c-48a6-98b5-5e907e2a59e9 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:09.989026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74334264-3fe7-4a59-8e07-3ca8d180c5fa None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.006701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:10.016602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c7200b-3448-424e-a03f-c3c3f74345d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.017242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c7200b-3448-424e-a03f-c3c3f74345d6 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.017521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c7200b-3448-424e-a03f-c3c3f74345d6 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.017813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c7200b-3448-424e-a03f-c3c3f74345d6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.027834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.028473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.028757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.028878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c7200b-3448-424e-a03f-c3c3f74345d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:10.029028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.029689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] RBAC: Adding query filter params (name=member) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:10.029992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.031374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-b0c7200b-3448-424e-a03f-c3c3f74345d6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:10.034222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:10.035140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:10.036013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 32/65] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:37:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:10.037133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c7200b-3448-424e-a03f-c3c3f74345d6 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.046312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 34/66] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.046312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.116607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:10.141100 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c7200b-3448-424e-a03f-c3c3f74345d6 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.156024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796bc805-0ce0-4783-aa76-1b8d5a70c855 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.176770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a13d99-89e4-4c0a-9a7e-eaf9baf85c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.177377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a13d99-89e4-4c0a-9a7e-eaf9baf85c80 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.177659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a13d99-89e4-4c0a-9a7e-eaf9baf85c80 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.177930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a13d99-89e4-4c0a-9a7e-eaf9baf85c80 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.178591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a13d99-89e4-4c0a-9a7e-eaf9baf85c80 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:10.178900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a13d99-89e4-4c0a-9a7e-eaf9baf85c80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.180323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a13d99-89e4-4c0a-9a7e-eaf9baf85c80 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.193655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 33/67] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.195580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9946baab-0c7c-4bf6-b684-d14ccd92b723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.196251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9946baab-0c7c-4bf6-b684-d14ccd92b723 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.196571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9946baab-0c7c-4bf6-b684-d14ccd92b723 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.196907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9946baab-0c7c-4bf6-b684-d14ccd92b723 None admin] PATH_INFO: `/v3/roles/member` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.197981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2009e61f-7474-4ecc-8175-87c5c2c34723 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.198304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2009e61f-7474-4ecc-8175-87c5c2c34723 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.198589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2009e61f-7474-4ecc-8175-87c5c2c34723 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.200003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 34/68] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:10.201218 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2009e61f-7474-4ecc-8175-87c5c2c34723 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.208383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9946baab-0c7c-4bf6-b684-d14ccd92b723 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-9946baab-0c7c-4bf6-b684-d14ccd92b723 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:10.213985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:10.214887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:10.215663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 35/69] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:10.216472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9946baab-0c7c-4bf6-b684-d14ccd92b723 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.258648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:10.312154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2009e61f-7474-4ecc-8175-87c5c2c34723 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.322922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9946baab-0c7c-4bf6-b684-d14ccd92b723 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.347174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.347787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.348071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.348376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.349038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:10.349343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.350738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.353846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:10.361759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 35/70] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.362684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.371179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6245dd22-6c0b-4bfa-8b08-3351fcccd62f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.371834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6245dd22-6c0b-4bfa-8b08-3351fcccd62f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.372111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6245dd22-6c0b-4bfa-8b08-3351fcccd62f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.372413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6245dd22-6c0b-4bfa-8b08-3351fcccd62f None admin] PATH_INFO: `/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.389189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6245dd22-6c0b-4bfa-8b08-3351fcccd62f None admin] RBAC: Authorizing `identity:get_user(user_id=97b89b4a23fb40a2bb9cfec9d4653ce7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.390663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6245dd22-6c0b-4bfa-8b08-3351fcccd62f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.392514 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 36/71] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7 => generated 275 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.393666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd7dd47-3cbc-4e89-b930-06ea63534b85 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.393952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd7dd47-3cbc-4e89-b930-06ea63534b85 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.394226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd7dd47-3cbc-4e89-b930-06ea63534b85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.413847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-8fd7dd47-3cbc-4e89-b930-06ea63534b85 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:10.415082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8fd7dd47-3cbc-4e89-b930-06ea63534b85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:10.484653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdc3020-7e97-4418-939b-a42b568265e6 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.494685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 37/72] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:10.496049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd7dd47-3cbc-4e89-b930-06ea63534b85 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.530739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e64a0-14f7-4f39-8d31-940f5cf41e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.531416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:10.531875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e64a0-14f7-4f39-8d31-940f5cf41e88 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.532712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e64a0-14f7-4f39-8d31-940f5cf41e88 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.532820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e64a0-14f7-4f39-8d31-940f5cf41e88 None admin] PATH_INFO: `/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.534264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483e64a0-14f7-4f39-8d31-940f5cf41e88 None admin] RBAC: Authorizing `identity:get_user(user_id=97b89b4a23fb40a2bb9cfec9d4653ce7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.536256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483e64a0-14f7-4f39-8d31-940f5cf41e88 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.538309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 36/73] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7 => generated 275 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.539240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e64a0-14f7-4f39-8d31-940f5cf41e88 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.616850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd7dd47-3cbc-4e89-b930-06ea63534b85 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.623056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:10.658043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.658668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.658949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.659225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.659888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] RBAC: Adding query filter params (name=member) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:10.660231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.661612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.662540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e64a0-14f7-4f39-8d31-940f5cf41e88 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.676179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 38/74] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.676179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.718733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f62c3c8-185e-41f1-ba56-56c03fa566e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.719465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f62c3c8-185e-41f1-ba56-56c03fa566e4 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.719994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f62c3c8-185e-41f1-ba56-56c03fa566e4 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.720378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f62c3c8-185e-41f1-ba56-56c03fa566e4 None admin] PATH_INFO: `/v3/users/0bb4ebf18f65413bbe4945aec1caead8` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.721653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f62c3c8-185e-41f1-ba56-56c03fa566e4 None admin] RBAC: Authorizing `identity:get_user(user_id=0bb4ebf18f65413bbe4945aec1caead8)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.723108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f62c3c8-185e-41f1-ba56-56c03fa566e4 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.724735 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 37/75] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/users/0bb4ebf18f65413bbe4945aec1caead8 => generated 247 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.730343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f62c3c8-185e-41f1-ba56-56c03fa566e4 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.775601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:10.788942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f591bdbe-95f4-4040-b9e9-2acad02f0a23 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.825031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e8eea2-ad65-483c-95a9-c704e4d0f118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.825663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e8eea2-ad65-483c-95a9-c704e4d0f118 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.825939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e8eea2-ad65-483c-95a9-c704e4d0f118 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.826210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e8eea2-ad65-483c-95a9-c704e4d0f118 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.827344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e8eea2-ad65-483c-95a9-c704e4d0f118 None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.828815 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e8eea2-ad65-483c-95a9-c704e4d0f118 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.830461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 39/76] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.832546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e8eea2-ad65-483c-95a9-c704e4d0f118 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.839111 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f62c3c8-185e-41f1-ba56-56c03fa566e4 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.875007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43898eac-9a23-40c1-9eda-18804df88243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.875566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43898eac-9a23-40c1-9eda-18804df88243 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.875854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43898eac-9a23-40c1-9eda-18804df88243 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.876166 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43898eac-9a23-40c1-9eda-18804df88243 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.877326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43898eac-9a23-40c1-9eda-18804df88243 None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.878805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43898eac-9a23-40c1-9eda-18804df88243 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.880501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 38/77] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.881271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43898eac-9a23-40c1-9eda-18804df88243 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:10.883900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:10.946481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e8eea2-ad65-483c-95a9-c704e4d0f118 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.984547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e3c526-ee78-4f2e-8322-0d188c2bda59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:10.985093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3c526-ee78-4f2e-8322-0d188c2bda59 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:10.985376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3c526-ee78-4f2e-8322-0d188c2bda59 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:10.985478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43898eac-9a23-40c1-9eda-18804df88243 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:10.985650 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3c526-ee78-4f2e-8322-0d188c2bda59 None admin] PATH_INFO: `/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:10.989194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e3c526-ee78-4f2e-8322-0d188c2bda59 None admin] RBAC: Authorizing `identity:get_user(user_id=97b89b4a23fb40a2bb9cfec9d4653ce7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:10.989194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e3c526-ee78-4f2e-8322-0d188c2bda59 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:10.991311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 40/78] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7 => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:10.991311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e3c526-ee78-4f2e-8322-0d188c2bda59 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:11.020144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:11.026600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d75a76-4bd7-4e6b-8de3-208d76c1408d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:11.027750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d75a76-4bd7-4e6b-8de3-208d76c1408d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:11.028031 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d75a76-4bd7-4e6b-8de3-208d76c1408d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:11.028349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d75a76-4bd7-4e6b-8de3-208d76c1408d None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:11.029526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d75a76-4bd7-4e6b-8de3-208d76c1408d None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:11.031005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d75a76-4bd7-4e6b-8de3-208d76c1408d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:11.032675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 39/79] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:11.033417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d75a76-4bd7-4e6b-8de3-208d76c1408d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:11.104201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e3c526-ee78-4f2e-8322-0d188c2bda59 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:11.109346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:11.142144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ce38ba-7cad-492f-81f8-8803b34a524c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:11.142704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ce38ba-7cad-492f-81f8-8803b34a524c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:11.142996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ce38ba-7cad-492f-81f8-8803b34a524c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:11.143275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ce38ba-7cad-492f-81f8-8803b34a524c None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:11.143583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d75a76-4bd7-4e6b-8de3-208d76c1408d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:11.144478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ce38ba-7cad-492f-81f8-8803b34a524c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:11.145982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ce38ba-7cad-492f-81f8-8803b34a524c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:11.147717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 41/80] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:11.148541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ce38ba-7cad-492f-81f8-8803b34a524c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:11.179850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38343d14-93cb-4f8f-9994-551c2f9862d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:11.180629 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38343d14-93cb-4f8f-9994-551c2f9862d2 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:11.180986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38343d14-93cb-4f8f-9994-551c2f9862d2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:11.181298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38343d14-93cb-4f8f-9994-551c2f9862d2 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b/users/97b89b4a23fb40a2bb9cfec9d4653ce7/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:11.183396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38343d14-93cb-4f8f-9994-551c2f9862d2 None admin] RBAC: Authorizing `identity:create_grant(project_id=cf4d543d56bb4a978f03e1af565b700b, user_id=97b89b4a23fb40a2bb9cfec9d4653ce7, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:11.184945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38343d14-93cb-4f8f-9994-551c2f9862d2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:11.201186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 40/81] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:11 2022] PUT /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b/users/97b89b4a23fb40a2bb9cfec9d4653ce7/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:11.202272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38343d14-93cb-4f8f-9994-551c2f9862d2 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:11.260949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:11.273386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ce38ba-7cad-492f-81f8-8803b34a524c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:11.317677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f971ba-e52c-47f6-a408-c46a92ce00f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:11.318733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f971ba-e52c-47f6-a408-c46a92ce00f9 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:11.319151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f971ba-e52c-47f6-a408-c46a92ce00f9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:11.319605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f971ba-e52c-47f6-a408-c46a92ce00f9 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b/users/97b89b4a23fb40a2bb9cfec9d4653ce7/roles/9b956c43ef364a70bf8263ba4314dce7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:11.322456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f971ba-e52c-47f6-a408-c46a92ce00f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=cf4d543d56bb4a978f03e1af565b700b, user_id=97b89b4a23fb40a2bb9cfec9d4653ce7, role_id=9b956c43ef364a70bf8263ba4314dce7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:11.324174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f971ba-e52c-47f6-a408-c46a92ce00f9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:11.341503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 42/82] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:11 2022] PUT /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b/users/97b89b4a23fb40a2bb9cfec9d4653ce7/roles/9b956c43ef364a70bf8263ba4314dce7 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:11.342034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f971ba-e52c-47f6-a408-c46a92ce00f9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:11.359290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38343d14-93cb-4f8f-9994-551c2f9862d2 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:11.410512 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:11.421106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbbcfe8-1b6d-4e63-9c5c-ce7b2f17fb39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:11.422232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbbcfe8-1b6d-4e63-9c5c-ce7b2f17fb39 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:11.422527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbbcfe8-1b6d-4e63-9c5c-ce7b2f17fb39 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:11.422810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbbcfe8-1b6d-4e63-9c5c-ce7b2f17fb39 None admin] PATH_INFO: `/v3/projects/2de03ad1e1464b40810a38cc5df8f969` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:11.424115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbbcfe8-1b6d-4e63-9c5c-ce7b2f17fb39 None admin] RBAC: Authorizing `identity:get_project(project_id=2de03ad1e1464b40810a38cc5df8f969)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:11.425801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbbcfe8-1b6d-4e63-9c5c-ce7b2f17fb39 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:11.427828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 41/83] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:11 2022] GET /identity/v3/projects/2de03ad1e1464b40810a38cc5df8f969 => generated 312 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:11.428698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbbcfe8-1b6d-4e63-9c5c-ce7b2f17fb39 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:11.513501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f971ba-e52c-47f6-a408-c46a92ce00f9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:11.568264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba73e3bc-a860-4b82-a372-159414f7a5a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:11.569320 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba73e3bc-a860-4b82-a372-159414f7a5a4 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:11.569829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba73e3bc-a860-4b82-a372-159414f7a5a4 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:11.569933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbbcfe8-1b6d-4e63-9c5c-ce7b2f17fb39 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:11.570425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba73e3bc-a860-4b82-a372-159414f7a5a4 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b/users/0bb4ebf18f65413bbe4945aec1caead8/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:11.573254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba73e3bc-a860-4b82-a372-159414f7a5a4 None admin] RBAC: Authorizing `identity:create_grant(project_id=cf4d543d56bb4a978f03e1af565b700b, user_id=0bb4ebf18f65413bbe4945aec1caead8, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:11.576056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba73e3bc-a860-4b82-a372-159414f7a5a4 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:11.598571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 43/84] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:11 2022] PUT /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b/users/0bb4ebf18f65413bbe4945aec1caead8/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:11.599994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba73e3bc-a860-4b82-a372-159414f7a5a4 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:11.623291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:11.624955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104bef3f-a650-4b81-845c-4648f2d392d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:11.627007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104bef3f-a650-4b81-845c-4648f2d392d7 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:11.627732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104bef3f-a650-4b81-845c-4648f2d392d7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:11.628255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104bef3f-a650-4b81-845c-4648f2d392d7 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:11.630007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104bef3f-a650-4b81-845c-4648f2d392d7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:11.631981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104bef3f-a650-4b81-845c-4648f2d392d7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:11.675644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 42/85] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:11 2022] POST /identity/v3/users => generated 289 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:11.757527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba73e3bc-a860-4b82-a372-159414f7a5a4 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:11.800552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8963b5-467c-451f-b2ab-c9f57f7b65f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:11.801591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8963b5-467c-451f-b2ab-c9f57f7b65f9 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:11.802052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8963b5-467c-451f-b2ab-c9f57f7b65f9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:11.802552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8963b5-467c-451f-b2ab-c9f57f7b65f9 None admin] PATH_INFO: `/v3/projects/2de03ad1e1464b40810a38cc5df8f969/users/97b89b4a23fb40a2bb9cfec9d4653ce7/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:11.809167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8963b5-467c-451f-b2ab-c9f57f7b65f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=2de03ad1e1464b40810a38cc5df8f969, user_id=97b89b4a23fb40a2bb9cfec9d4653ce7, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:11.811059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8963b5-467c-451f-b2ab-c9f57f7b65f9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:11.838287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 44/86] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:11 2022] PUT /identity/v3/projects/2de03ad1e1464b40810a38cc5df8f969/users/97b89b4a23fb40a2bb9cfec9d4653ce7/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:14.012433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a69bc-7b68-4011-9d8f-1044d5a20abd None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:14.012964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a69bc-7b68-4011-9d8f-1044d5a20abd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:14.013424 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a69bc-7b68-4011-9d8f-1044d5a20abd None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:14.015924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 43/87] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:14.022690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026337c7-b2a7-4635-acf1-bd95b8d7018a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:14.023274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026337c7-b2a7-4635-acf1-bd95b8d7018a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:14.023692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026337c7-b2a7-4635-acf1-bd95b8d7018a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:14.069735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-026337c7-b2a7-4635-acf1-bd95b8d7018a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:14.072226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-026337c7-b2a7-4635-acf1-bd95b8d7018a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:14.222143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 45/88] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:14.773666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873a69bc-7b68-4011-9d8f-1044d5a20abd None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:14.867502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:14.948987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873a69bc-7b68-4011-9d8f-1044d5a20abd None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:15.001553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283ee205-1e75-4e4d-9675-95526e1ce83a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:15.002223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283ee205-1e75-4e4d-9675-95526e1ce83a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:15.002513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283ee205-1e75-4e4d-9675-95526e1ce83a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:15.002796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283ee205-1e75-4e4d-9675-95526e1ce83a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:15.009940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52fdc31-c404-4f0f-a449-3efab9df319b None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:15.010250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52fdc31-c404-4f0f-a449-3efab9df319b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:15.010724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52fdc31-c404-4f0f-a449-3efab9df319b None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:15.012753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 46/89] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:15.026371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f2e02-c76b-418f-a03d-aaed24150b02 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:15.026967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f2e02-c76b-418f-a03d-aaed24150b02 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:15.026967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f2e02-c76b-418f-a03d-aaed24150b02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:15.028563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283ee205-1e75-4e4d-9675-95526e1ce83a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-283ee205-1e75-4e4d-9675-95526e1ce83a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:15.036005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:15.036987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:15.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:15.039452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 44/90] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:37:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:15.044655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:15.047194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283ee205-1e75-4e4d-9675-95526e1ce83a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:15.069198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ad9f2e02-c76b-418f-a03d-aaed24150b02 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:15.071099 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ad9f2e02-c76b-418f-a03d-aaed24150b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:15.244626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 47/91] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:15.247061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9066e43e-912e-4023-b312-c2baeb966f50 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:15.247453 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9066e43e-912e-4023-b312-c2baeb966f50 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:15.247620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9066e43e-912e-4023-b312-c2baeb966f50 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:15.253547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 48/92] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:15 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:15.262115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4717b5-a0fb-426b-8d6e-f76027957056 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:15.262479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4717b5-a0fb-426b-8d6e-f76027957056 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:15.262713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4717b5-a0fb-426b-8d6e-f76027957056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:15.334315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fc4717b5-a0fb-426b-8d6e-f76027957056 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:15.335448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fc4717b5-a0fb-426b-8d6e-f76027957056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:15.420036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283ee205-1e75-4e4d-9675-95526e1ce83a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:15.476829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:15.557156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:15.558276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:15.558772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:15.559275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:15.560300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:15.560820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:15.562473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:15.573423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 49/93] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:15.580169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159b1116-c81e-4ec9-8204-d1643943212f None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:15.580169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159b1116-c81e-4ec9-8204-d1643943212f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:15.580169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159b1116-c81e-4ec9-8204-d1643943212f None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:15.580169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 50/94] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:15.585130 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043a73e5-07fe-45af-b299-14b29c0a2831 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:15.585374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043a73e5-07fe-45af-b299-14b29c0a2831 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:15.585659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043a73e5-07fe-45af-b299-14b29c0a2831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:15.587862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 45/95] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:37:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 541 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:15.596155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:15.607568 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-043a73e5-07fe-45af-b299-14b29c0a2831 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:15.608783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-043a73e5-07fe-45af-b299-14b29c0a2831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:15.747138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:15.783674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 51/96] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:15.787430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043a73e5-07fe-45af-b299-14b29c0a2831 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:15.939494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7cd231-8624-44bf-b2d3-82c1de99d970 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.035396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf528a6-82d2-4f54-962a-44c089fcc687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:16.036709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf528a6-82d2-4f54-962a-44c089fcc687 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.036996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf528a6-82d2-4f54-962a-44c089fcc687 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.037272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf528a6-82d2-4f54-962a-44c089fcc687 None admin] PATH_INFO: `/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.039051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf528a6-82d2-4f54-962a-44c089fcc687 None admin] RBAC: Authorizing `identity:get_user(user_id=97b89b4a23fb40a2bb9cfec9d4653ce7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:16.040663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf528a6-82d2-4f54-962a-44c089fcc687 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:16.043223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 46/97] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:15 2022] GET /identity/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7 => generated 275 bytes in 448 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:16.048619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf528a6-82d2-4f54-962a-44c089fcc687 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:16.086028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043a73e5-07fe-45af-b299-14b29c0a2831 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.120298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:16.137454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58542ead-5d6f-4035-a124-e939a936d34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:16.138114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58542ead-5d6f-4035-a124-e939a936d34d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.138390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58542ead-5d6f-4035-a124-e939a936d34d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.138716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58542ead-5d6f-4035-a124-e939a936d34d None admin] PATH_INFO: `/v3/roles/member` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.153337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58542ead-5d6f-4035-a124-e939a936d34d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-58542ead-5d6f-4035-a124-e939a936d34d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:16.158452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:16.159592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:16.161086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 52/98] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 376 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:16.162108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34f87a1-6858-4e8b-b55e-09734cc26695 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.162108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34f87a1-6858-4e8b-b55e-09734cc26695 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.162108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34f87a1-6858-4e8b-b55e-09734cc26695 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.165210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 53/99] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:16.165210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34f87a1-6858-4e8b-b55e-09734cc26695 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:16.197547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf528a6-82d2-4f54-962a-44c089fcc687 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.236570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ef9ec1-e3f3-476d-980e-1df7a74ea6d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:16.237246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ef9ec1-e3f3-476d-980e-1df7a74ea6d1 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.237595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ef9ec1-e3f3-476d-980e-1df7a74ea6d1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.237796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ef9ec1-e3f3-476d-980e-1df7a74ea6d1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.244406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:16.248775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ef9ec1-e3f3-476d-980e-1df7a74ea6d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-b0ef9ec1-e3f3-476d-980e-1df7a74ea6d1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:16.254153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:16.255235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:16.256025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 47/100] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:16.256701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2376b5f9-5818-4ab6-ac3f-416b6c4b62ea None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.257632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2376b5f9-5818-4ab6-ac3f-416b6c4b62ea None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.257632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2376b5f9-5818-4ab6-ac3f-416b6c4b62ea None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.258529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 48/101] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:16.263513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2376b5f9-5818-4ab6-ac3f-416b6c4b62ea None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:16.311810 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34f87a1-6858-4e8b-b55e-09734cc26695 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.350845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ae3bac-2877-47ff-aadf-599aa9e70012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:16.351505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ae3bac-2877-47ff-aadf-599aa9e70012 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.351758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ae3bac-2877-47ff-aadf-599aa9e70012 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.352033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ae3bac-2877-47ff-aadf-599aa9e70012 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.377700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2376b5f9-5818-4ab6-ac3f-416b6c4b62ea None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.380885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:16.420228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190e4b10-6e18-4eb6-b5b3-d55669c8deec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:16.420887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190e4b10-6e18-4eb6-b5b3-d55669c8deec None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.421200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190e4b10-6e18-4eb6-b5b3-d55669c8deec None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.421447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190e4b10-6e18-4eb6-b5b3-d55669c8deec None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.433637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ae3bac-2877-47ff-aadf-599aa9e70012 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:16.434856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190e4b10-6e18-4eb6-b5b3-d55669c8deec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:16.435172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ae3bac-2877-47ff-aadf-599aa9e70012 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-190e4b10-6e18-4eb6-b5b3-d55669c8deec None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:16.439982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:16.441060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 49/102] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:37:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:16.442098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190e4b10-6e18-4eb6-b5b3-d55669c8deec None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:16.498860 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:16.554302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 54/103] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:16.555609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0459c0-30e1-4bee-aaff-072813d0c6cc None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.557681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0459c0-30e1-4bee-aaff-072813d0c6cc None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.557681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0459c0-30e1-4bee-aaff-072813d0c6cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.581300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0a0459c0-30e1-4bee-aaff-072813d0c6cc None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:16.582804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0a0459c0-30e1-4bee-aaff-072813d0c6cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:16.582921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190e4b10-6e18-4eb6-b5b3-d55669c8deec None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.621822 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:16.622439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.622722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.623002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.623688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] RBAC: Adding query filter params (name=member) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:16.623993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:16.625478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:16.634648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 50/104] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:16.635516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:16.666444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 55/105] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:16.667936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca3bcd-b0d6-4b8e-bf76-ec0ae2ba58df None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.668327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca3bcd-b0d6-4b8e-bf76-ec0ae2ba58df None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.668624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca3bcd-b0d6-4b8e-bf76-ec0ae2ba58df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.677491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:16.690573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-42ca3bcd-b0d6-4b8e-bf76-ec0ae2ba58df None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:16.691802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-42ca3bcd-b0d6-4b8e-bf76-ec0ae2ba58df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:16.779173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 56/106] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:16.781407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1ad07e-ae52-480a-9aec-8c86b55536fc None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.781783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ca3bcd-b0d6-4b8e-bf76-ec0ae2ba58df None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:16.807574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:16.827103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:16.827819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.828206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.828576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.829370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] RBAC: Adding query filter params (name=member) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:16.829734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:16.831225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:16.845128 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 51/107] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:16.850826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:16.932008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ca3bcd-b0d6-4b8e-bf76-ec0ae2ba58df None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.959627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:16.982500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:16.983377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd2947b-78c6-4e5a-affb-3408e0d3bd32 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:16.983534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:16.984103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:16.984597 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:16.988707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:16.988707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:16.988707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.003172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 57/108] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:17.003172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.042363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff4bb1b-1839-4a2d-8578-e9406df9f308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.042902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4bb1b-1839-4a2d-8578-e9406df9f308 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.043286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4bb1b-1839-4a2d-8578-e9406df9f308 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.043627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4bb1b-1839-4a2d-8578-e9406df9f308 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.045004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff4bb1b-1839-4a2d-8578-e9406df9f308 None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:17.046593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff4bb1b-1839-4a2d-8578-e9406df9f308 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.048647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 52/109] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:16 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:17.049695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff4bb1b-1839-4a2d-8578-e9406df9f308 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.105530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:17.141127 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2465b5-bc12-4134-b156-127ddc139845 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:17.180670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd3584-1e62-4107-9113-8d65cff59ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.182179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd3584-1e62-4107-9113-8d65cff59ed6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.182465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd3584-1e62-4107-9113-8d65cff59ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.182746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd3584-1e62-4107-9113-8d65cff59ed6 None admin] PATH_INFO: `/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.183926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addd3584-1e62-4107-9113-8d65cff59ed6 None admin] RBAC: Authorizing `identity:get_user(user_id=97b89b4a23fb40a2bb9cfec9d4653ce7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:17.185445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addd3584-1e62-4107-9113-8d65cff59ed6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.187222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 58/110] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:16 2022] GET /identity/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7 => generated 275 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:17.188288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd3584-1e62-4107-9113-8d65cff59ed6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.197017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff4bb1b-1839-4a2d-8578-e9406df9f308 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:17.235540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729c2671-9e0f-4fc8-936c-c93fa37c9f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.236196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729c2671-9e0f-4fc8-936c-c93fa37c9f10 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.236527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729c2671-9e0f-4fc8-936c-c93fa37c9f10 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.236809 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729c2671-9e0f-4fc8-936c-c93fa37c9f10 None admin] PATH_INFO: `/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.238194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729c2671-9e0f-4fc8-936c-c93fa37c9f10 None admin] RBAC: Authorizing `identity:get_user(user_id=97b89b4a23fb40a2bb9cfec9d4653ce7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:17.239759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729c2671-9e0f-4fc8-936c-c93fa37c9f10 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.241679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 53/111] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/users/97b89b4a23fb40a2bb9cfec9d4653ce7 => generated 275 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:17.242466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729c2671-9e0f-4fc8-936c-c93fa37c9f10 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.271731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:17.328504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd3584-1e62-4107-9113-8d65cff59ed6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:17.356346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729c2671-9e0f-4fc8-936c-c93fa37c9f10 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:17.368923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865a7a11-c7c5-47dd-b113-323da3f7dfb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.369743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865a7a11-c7c5-47dd-b113-323da3f7dfb4 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.370027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865a7a11-c7c5-47dd-b113-323da3f7dfb4 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.370301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865a7a11-c7c5-47dd-b113-323da3f7dfb4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.382109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865a7a11-c7c5-47dd-b113-323da3f7dfb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-865a7a11-c7c5-47dd-b113-323da3f7dfb4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:17.389184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:17.391140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 59/112] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:17.392342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865a7a11-c7c5-47dd-b113-323da3f7dfb4 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.394632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:17.404830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240355d1-62f0-44ab-9383-7bdcf41c4313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.405477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240355d1-62f0-44ab-9383-7bdcf41c4313 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.405770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240355d1-62f0-44ab-9383-7bdcf41c4313 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.406057 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240355d1-62f0-44ab-9383-7bdcf41c4313 None admin] PATH_INFO: `/v3/users/0bb4ebf18f65413bbe4945aec1caead8` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.407226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240355d1-62f0-44ab-9383-7bdcf41c4313 None admin] RBAC: Authorizing `identity:get_user(user_id=0bb4ebf18f65413bbe4945aec1caead8)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:17.410936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240355d1-62f0-44ab-9383-7bdcf41c4313 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.413577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 54/113] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/users/0bb4ebf18f65413bbe4945aec1caead8 => generated 247 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:17.413577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240355d1-62f0-44ab-9383-7bdcf41c4313 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.512510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865a7a11-c7c5-47dd-b113-323da3f7dfb4 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:17.520098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:17.532991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240355d1-62f0-44ab-9383-7bdcf41c4313 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:17.552464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc14bc17-e4a7-4fef-ac58-4d32986e48be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.553115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc14bc17-e4a7-4fef-ac58-4d32986e48be None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.553396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc14bc17-e4a7-4fef-ac58-4d32986e48be None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.553669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc14bc17-e4a7-4fef-ac58-4d32986e48be None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.554837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc14bc17-e4a7-4fef-ac58-4d32986e48be None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:17.556890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc14bc17-e4a7-4fef-ac58-4d32986e48be None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.558939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 60/114] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:17.559774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc14bc17-e4a7-4fef-ac58-4d32986e48be None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.570405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.571034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.571360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.571592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.572950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] RBAC: Adding query filter params (role.id=9b956c43ef364a70bf8263ba4314dce7, scope.project.id=cf4d543d56bb4a978f03e1af565b700b, user.id=97b89b4a23fb40a2bb9cfec9d4653ce7) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:17.573417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:17.574921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.583221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 55/115] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/role_assignments?user.id=97b89b4a23fb40a2bb9cfec9d4653ce7&scope.project.id=cf4d543d56bb4a978f03e1af565b700b&role.id=9b956c43ef364a70bf8263ba4314dce7 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:17.584100 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.648352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:17.692446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc14bc17-e4a7-4fef-ac58-4d32986e48be None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:17.732481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594844a-19df-48d7-ba1d-a76ddd67d0c3 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:17.738659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e1521-158d-4f64-83b1-e3dbeee6bace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.739673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e1521-158d-4f64-83b1-e3dbeee6bace None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.740827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e1521-158d-4f64-83b1-e3dbeee6bace None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.741624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e1521-158d-4f64-83b1-e3dbeee6bace None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.773685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f466ec1-0039-4f7a-8794-a93c64c8acdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:17.774313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f466ec1-0039-4f7a-8794-a93c64c8acdf None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:17.774626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f466ec1-0039-4f7a-8794-a93c64c8acdf None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:17.774897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f466ec1-0039-4f7a-8794-a93c64c8acdf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:17.805679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:17.833590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66e1521-158d-4f64-83b1-e3dbeee6bace None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:17.834976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66e1521-158d-4f64-83b1-e3dbeee6bace None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.855548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f466ec1-0039-4f7a-8794-a93c64c8acdf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:17.857077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f466ec1-0039-4f7a-8794-a93c64c8acdf None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:17.906466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:17.939390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 61/116] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:17.940278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e1521-158d-4f64-83b1-e3dbeee6bace None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:17.964617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 56/117] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:17.964617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f466ec1-0039-4f7a-8794-a93c64c8acdf None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.023828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:18.054106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e1521-158d-4f64-83b1-e3dbeee6bace None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.069963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f466ec1-0039-4f7a-8794-a93c64c8acdf None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.091638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.092183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.092471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.092742 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.093404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:18.093712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.095096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.104592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca66134-fb21-4bc4-8d93-71083edc9896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.104766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 62/118] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:18.105265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca66134-fb21-4bc4-8d93-71083edc9896 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.105501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca66134-fb21-4bc4-8d93-71083edc9896 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.105595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.106022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca66134-fb21-4bc4-8d93-71083edc9896 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.138400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:18.179386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca66134-fb21-4bc4-8d93-71083edc9896 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.180888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca66134-fb21-4bc4-8d93-71083edc9896 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.208871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132e0a99-d5f5-408d-a133-ce116e1f2bc9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.237075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:18.244426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d29c3b-ad5f-49bc-8bd1-ab23839ecd9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.245238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d29c3b-ad5f-49bc-8bd1-ab23839ecd9c None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.245522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d29c3b-ad5f-49bc-8bd1-ab23839ecd9c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.245800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d29c3b-ad5f-49bc-8bd1-ab23839ecd9c None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.246464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d29c3b-ad5f-49bc-8bd1-ab23839ecd9c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.247917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d29c3b-ad5f-49bc-8bd1-ab23839ecd9c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.278200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 63/119] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:37:18 2022] POST /identity/v3/projects => generated 302 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:18.281646 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d29c3b-ad5f-49bc-8bd1-ab23839ecd9c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.289772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 57/120] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:18.290666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca66134-fb21-4bc4-8d93-71083edc9896 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.385531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:18.409431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d29c3b-ad5f-49bc-8bd1-ab23839ecd9c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.425815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca66134-fb21-4bc4-8d93-71083edc9896 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.450941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe9656-41b7-43d3-977f-930df3435962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.451623 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe9656-41b7-43d3-977f-930df3435962 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.451900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe9656-41b7-43d3-977f-930df3435962 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.452204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe9656-41b7-43d3-977f-930df3435962 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.453422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fe9656-41b7-43d3-977f-930df3435962 None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.454893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fe9656-41b7-43d3-977f-930df3435962 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.456658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 64/121] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:18 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:18.461678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe9656-41b7-43d3-977f-930df3435962 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.466542 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a544365-8bfd-4131-b3b4-ba5e4b1bc710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.467188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a544365-8bfd-4131-b3b4-ba5e4b1bc710 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.467484 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a544365-8bfd-4131-b3b4-ba5e4b1bc710 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.467743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a544365-8bfd-4131-b3b4-ba5e4b1bc710 None admin] PATH_INFO: `/v3/projects/2de03ad1e1464b40810a38cc5df8f969` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.468960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a544365-8bfd-4131-b3b4-ba5e4b1bc710 None admin] RBAC: Authorizing `identity:get_project(project_id=2de03ad1e1464b40810a38cc5df8f969)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.470418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a544365-8bfd-4131-b3b4-ba5e4b1bc710 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.472249 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 58/122] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:18 2022] GET /identity/v3/projects/2de03ad1e1464b40810a38cc5df8f969 => generated 312 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:18.473224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a544365-8bfd-4131-b3b4-ba5e4b1bc710 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.516106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:18.579403 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe9656-41b7-43d3-977f-930df3435962 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.617540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdf8402-6bd8-4960-a6bc-a23f6930f5bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.618186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdf8402-6bd8-4960-a6bc-a23f6930f5bd None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.618488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdf8402-6bd8-4960-a6bc-a23f6930f5bd None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.619342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdf8402-6bd8-4960-a6bc-a23f6930f5bd None admin] PATH_INFO: `/v3/users/fb4de295c5984827924792596b7cf6f9` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.634459 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdf8402-6bd8-4960-a6bc-a23f6930f5bd None admin] RBAC: Authorizing `identity:get_user(user_id=fb4de295c5984827924792596b7cf6f9)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.635999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdf8402-6bd8-4960-a6bc-a23f6930f5bd None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.642336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 65/123] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:18 2022] GET /identity/v3/users/fb4de295c5984827924792596b7cf6f9 => generated 289 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:18.642336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdf8402-6bd8-4960-a6bc-a23f6930f5bd None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.654169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:18.662813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a544365-8bfd-4131-b3b4-ba5e4b1bc710 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.710706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48eed7b3-5d05-45be-b171-54a00aced928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.711592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48eed7b3-5d05-45be-b171-54a00aced928 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.711987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48eed7b3-5d05-45be-b171-54a00aced928 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.713016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48eed7b3-5d05-45be-b171-54a00aced928 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.714639 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48eed7b3-5d05-45be-b171-54a00aced928 None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.716823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48eed7b3-5d05-45be-b171-54a00aced928 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.719560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 59/124] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:18 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:18.720514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48eed7b3-5d05-45be-b171-54a00aced928 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.764194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdf8402-6bd8-4960-a6bc-a23f6930f5bd None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.800285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.800891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.801173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.801447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.802170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] RBAC: Adding query filter params (role.id=05482458bad348e0be384b75bfd31eac, scope.project.id=cf4d543d56bb4a978f03e1af565b700b, user.id=97b89b4a23fb40a2bb9cfec9d4653ce7) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:18.802505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.803942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.810665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 66/125] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:18 2022] GET /identity/v3/role_assignments?user.id=97b89b4a23fb40a2bb9cfec9d4653ce7&scope.project.id=cf4d543d56bb4a978f03e1af565b700b&role.id=05482458bad348e0be384b75bfd31eac => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:18.811493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:18.879080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:18.893129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48eed7b3-5d05-45be-b171-54a00aced928 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:18.943763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:18.944738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:18.945210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:18.945659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:18.946703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] RBAC: Adding query filter params (role.id=05482458bad348e0be384b75bfd31eac, scope.project.id=2de03ad1e1464b40810a38cc5df8f969, user.id=97b89b4a23fb40a2bb9cfec9d4653ce7) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:18.947235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:18.948927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:18.950186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:18.959699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 60/126] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:18 2022] GET /identity/v3/role_assignments?user.id=97b89b4a23fb40a2bb9cfec9d4653ce7&scope.project.id=2de03ad1e1464b40810a38cc5df8f969&role.id=05482458bad348e0be384b75bfd31eac => generated 608 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:18.960958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:19.055531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caba8014-7223-437d-b28a-d51b4a8fd3fc None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:19.096225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e41187-ff3f-4183-8cff-f53286c9bd84 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:19.144818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:19.146214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:19.146774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:19.147277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:19.148347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=cf4d543d56bb4a978f03e1af565b700b, user.id=0bb4ebf18f65413bbe4945aec1caead8) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:19.148943 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:19.150794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:19.166600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 61/127] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:18 2022] GET /identity/v3/role_assignments?user.id=0bb4ebf18f65413bbe4945aec1caead8&scope.project.id=cf4d543d56bb4a978f03e1af565b700b&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 608 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:19.183210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11fa597-f295-4869-ab7f-a2f55ea545f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:19.185834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11fa597-f295-4869-ab7f-a2f55ea545f0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:19.186547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11fa597-f295-4869-ab7f-a2f55ea545f0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:19.187039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11fa597-f295-4869-ab7f-a2f55ea545f0 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:19.188929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11fa597-f295-4869-ab7f-a2f55ea545f0 None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:19.190812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11fa597-f295-4869-ab7f-a2f55ea545f0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:19.193177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 67/128] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:18 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 382 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:19.199681 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:19.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:19.337955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644ccb6-8b67-4bba-b739-7a0bd1f8efc9 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:19.380898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca195dd7-8314-4e46-b09d-39a66da270cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:19.381584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca195dd7-8314-4e46-b09d-39a66da270cf None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:19.381866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca195dd7-8314-4e46-b09d-39a66da270cf None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:19.382145 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca195dd7-8314-4e46-b09d-39a66da270cf None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b/users/fb4de295c5984827924792596b7cf6f9/roles/6d01ecb1b0554e219119728a79f21015` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:19.384477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca195dd7-8314-4e46-b09d-39a66da270cf None admin] RBAC: Authorizing `identity:create_grant(project_id=cf4d543d56bb4a978f03e1af565b700b, user_id=fb4de295c5984827924792596b7cf6f9, role_id=6d01ecb1b0554e219119728a79f21015)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:19.386029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca195dd7-8314-4e46-b09d-39a66da270cf None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:19.402164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 62/129] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:19 2022] PUT /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b/users/fb4de295c5984827924792596b7cf6f9/roles/6d01ecb1b0554e219119728a79f21015 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:37:20.625528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab20b8-a243-4fdd-804f-6ef10651fbe4 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:20.626093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab20b8-a243-4fdd-804f-6ef10651fbe4 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:20.626093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab20b8-a243-4fdd-804f-6ef10651fbe4 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:20.627639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 68/130] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:20.633275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b405b6a-f249-4570-9a99-40ff8b1d8620 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:20.633890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b405b6a-f249-4570-9a99-40ff8b1d8620 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:20.634385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b405b6a-f249-4570-9a99-40ff8b1d8620 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:20.654322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2b405b6a-f249-4570-9a99-40ff8b1d8620 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:20.655836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2b405b6a-f249-4570-9a99-40ff8b1d8620 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:20.748619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 63/131] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:21.028962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ab20b8-a243-4fdd-804f-6ef10651fbe4 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:21.169256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ab20b8-a243-4fdd-804f-6ef10651fbe4 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:21.205844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d768151b-e018-4a99-b470-4ee116016b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:21.206476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d768151b-e018-4a99-b470-4ee116016b77 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:21.206749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d768151b-e018-4a99-b470-4ee116016b77 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:21.207013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d768151b-e018-4a99-b470-4ee116016b77 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:21.208192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d768151b-e018-4a99-b470-4ee116016b77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:21.209758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d768151b-e018-4a99-b470-4ee116016b77 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:21.211549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 69/132] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:21.217815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b405b6a-f249-4570-9a99-40ff8b1d8620 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:21.231798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:21.329895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b405b6a-f249-4570-9a99-40ff8b1d8620 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:21.371232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8cb787-0b20-4445-9072-ea98a9a487ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:21.371926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8cb787-0b20-4445-9072-ea98a9a487ac None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:21.372389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8cb787-0b20-4445-9072-ea98a9a487ac None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:21.372672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8cb787-0b20-4445-9072-ea98a9a487ac None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:21.373458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8cb787-0b20-4445-9072-ea98a9a487ac None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:21.375053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8cb787-0b20-4445-9072-ea98a9a487ac None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:21.399422 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 64/133] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:21 2022] POST /identity/v3/users => generated 283 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:21.780917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256bbccc-279e-45e5-aa3b-b0fc9f80c963 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:21.781857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256bbccc-279e-45e5-aa3b-b0fc9f80c963 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:21.781857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256bbccc-279e-45e5-aa3b-b0fc9f80c963 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:21.783447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 70/134] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:21.790341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d91cc-f0ef-4343-8efd-6494c5f3ea7f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:21.791078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d91cc-f0ef-4343-8efd-6494c5f3ea7f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:21.791236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d91cc-f0ef-4343-8efd-6494c5f3ea7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:21.819200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-035d91cc-f0ef-4343-8efd-6494c5f3ea7f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:21.820515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-035d91cc-f0ef-4343-8efd-6494c5f3ea7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:21.900690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:21.909594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 65/135] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:22.263834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256bbccc-279e-45e5-aa3b-b0fc9f80c963 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:22.279154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:22.383977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256bbccc-279e-45e5-aa3b-b0fc9f80c963 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:22.426088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e5269-3b00-4fc1-ac21-5f8700fd0042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:22.426756 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e5269-3b00-4fc1-ac21-5f8700fd0042 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:22.427037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e5269-3b00-4fc1-ac21-5f8700fd0042 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:22.427308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e5269-3b00-4fc1-ac21-5f8700fd0042 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:22.440934 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09e5269-3b00-4fc1-ac21-5f8700fd0042 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-d09e5269-3b00-4fc1-ac21-5f8700fd0042 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:22.446585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:22.448190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:22.449084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 71/136] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:22.455324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035d91cc-f0ef-4343-8efd-6494c5f3ea7f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:22.586061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035d91cc-f0ef-4343-8efd-6494c5f3ea7f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:22.637286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:22.637937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:22.638224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:22.638504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:22.639220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] RBAC: Adding query filter params (name=reader) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:22.639533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:22.641060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:22.652865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 66/137] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:22.659092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e5269-3b00-4fc1-ac21-5f8700fd0042 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:22.753747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:22.796356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e5269-3b00-4fc1-ac21-5f8700fd0042 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:22.832150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1489a40c-4251-4302-9de6-fba326f5755f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:22.832789 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1489a40c-4251-4302-9de6-fba326f5755f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:22.833068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1489a40c-4251-4302-9de6-fba326f5755f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:22.833342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1489a40c-4251-4302-9de6-fba326f5755f None admin] PATH_INFO: `/v3/users/fb4de295c5984827924792596b7cf6f9` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:22.834595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1489a40c-4251-4302-9de6-fba326f5755f None admin] RBAC: Authorizing `identity:get_user(user_id=fb4de295c5984827924792596b7cf6f9)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:22.836156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1489a40c-4251-4302-9de6-fba326f5755f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:22.839120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 72/138] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:22 2022] GET /identity/v3/users/fb4de295c5984827924792596b7cf6f9 => generated 289 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:22.844419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:22.907361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:22.957290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88552ed7-32ec-4cad-821c-353bd61a2795 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:23.001495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1629a617-d83a-4b6d-a25a-e7b0d9abf85c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:23.002452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1629a617-d83a-4b6d-a25a-e7b0d9abf85c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:23.002951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1629a617-d83a-4b6d-a25a-e7b0d9abf85c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:23.003449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1629a617-d83a-4b6d-a25a-e7b0d9abf85c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:23.075937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1629a617-d83a-4b6d-a25a-e7b0d9abf85c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:23.077831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1629a617-d83a-4b6d-a25a-e7b0d9abf85c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:23.152163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:23.200895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 67/139] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:23.206955 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1489a40c-4251-4302-9de6-fba326f5755f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:23.321044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1489a40c-4251-4302-9de6-fba326f5755f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:23.364770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb73e30-2dee-4192-acf4-7f7fae91ac55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:23.365418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb73e30-2dee-4192-acf4-7f7fae91ac55 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:23.365695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb73e30-2dee-4192-acf4-7f7fae91ac55 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:23.365965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb73e30-2dee-4192-acf4-7f7fae91ac55 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:23.367390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb73e30-2dee-4192-acf4-7f7fae91ac55 None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:23.368922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb73e30-2dee-4192-acf4-7f7fae91ac55 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:23.371607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 73/140] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:23 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:23.377804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1629a617-d83a-4b6d-a25a-e7b0d9abf85c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:23.502120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1629a617-d83a-4b6d-a25a-e7b0d9abf85c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:23.543877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cb37fc-84aa-4966-b458-9ff63abfa469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:23.544550 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb37fc-84aa-4966-b458-9ff63abfa469 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:23.544889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb37fc-84aa-4966-b458-9ff63abfa469 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:23.545178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb37fc-84aa-4966-b458-9ff63abfa469 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:23.545925 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cb37fc-84aa-4966-b458-9ff63abfa469 None admin] RBAC: Adding query filter params (role.id=6d01ecb1b0554e219119728a79f21015, scope.project.id=cf4d543d56bb4a978f03e1af565b700b, user.id=fb4de295c5984827924792596b7cf6f9) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:23.546241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cb37fc-84aa-4966-b458-9ff63abfa469 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:23.547702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cb37fc-84aa-4966-b458-9ff63abfa469 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:23.555045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 68/141] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:23 2022] GET /identity/v3/role_assignments?user.id=fb4de295c5984827924792596b7cf6f9&scope.project.id=cf4d543d56bb4a978f03e1af565b700b&role.id=6d01ecb1b0554e219119728a79f21015 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:23.910613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c9851e-eab1-459e-8529-00a0e416cc38 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:23.911542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c9851e-eab1-459e-8529-00a0e416cc38 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:23.911998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c9851e-eab1-459e-8529-00a0e416cc38 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:23.913865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 74/142] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:23.919305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94fdff-9d1d-4536-b86b-303bb56f77e4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:23.919726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94fdff-9d1d-4536-b86b-303bb56f77e4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:23.920068 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94fdff-9d1d-4536-b86b-303bb56f77e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:23.938902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5e94fdff-9d1d-4536-b86b-303bb56f77e4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:23.940097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5e94fdff-9d1d-4536-b86b-303bb56f77e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:23.958397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:24.019594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 69/143] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:24.083424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af489124-861e-40ee-8276-10cad3cf403e None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.083705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af489124-861e-40ee-8276-10cad3cf403e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.084161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af489124-861e-40ee-8276-10cad3cf403e None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.085544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 75/144] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:24.091688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973666f9-f392-4835-8b67-e8910c6fad69 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.091989 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973666f9-f392-4835-8b67-e8910c6fad69 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.092278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973666f9-f392-4835-8b67-e8910c6fad69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.110525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-973666f9-f392-4835-8b67-e8910c6fad69 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:24.111719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-973666f9-f392-4835-8b67-e8910c6fad69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:24.176664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88596fb4-f6a7-4270-9866-e41014789fa6 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.176855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88596fb4-f6a7-4270-9866-e41014789fa6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.177128 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88596fb4-f6a7-4270-9866-e41014789fa6 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.178472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 76/145] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:37:24.179501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 70/146] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:24.185290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43904550-c8aa-415a-b42d-d0a11305d6f3 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.185575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43904550-c8aa-415a-b42d-d0a11305d6f3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.185851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43904550-c8aa-415a-b42d-d0a11305d6f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.198683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:24.204431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-43904550-c8aa-415a-b42d-d0a11305d6f3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:24.205614 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-43904550-c8aa-415a-b42d-d0a11305d6f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:24.273210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 77/147] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:24.293216 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973666f9-f392-4835-8b67-e8910c6fad69 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:24.374837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:24.394231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973666f9-f392-4835-8b67-e8910c6fad69 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:24.428900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984ba1d0-c327-4042-a892-0cb24036f551 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:24.429445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984ba1d0-c327-4042-a892-0cb24036f551 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.429775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984ba1d0-c327-4042-a892-0cb24036f551 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.430052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984ba1d0-c327-4042-a892-0cb24036f551 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.440319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984ba1d0-c327-4042-a892-0cb24036f551 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-984ba1d0-c327-4042-a892-0cb24036f551 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:24.445079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:24.445945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:24.446974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 71/148] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:37:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:24.450953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43904550-c8aa-415a-b42d-d0a11305d6f3 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:24.452323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984ba1d0-c327-4042-a892-0cb24036f551 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:24.566316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43904550-c8aa-415a-b42d-d0a11305d6f3 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:24.576406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984ba1d0-c327-4042-a892-0cb24036f551 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:24.605800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074d6c35-7945-4152-b964-1c2f9bc42eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:24.606439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074d6c35-7945-4152-b964-1c2f9bc42eb0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.606720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074d6c35-7945-4152-b964-1c2f9bc42eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.606990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074d6c35-7945-4152-b964-1c2f9bc42eb0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.609877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:24.619332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074d6c35-7945-4152-b964-1c2f9bc42eb0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:24.620064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:24.620713 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.620993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.621268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.621936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:24.622237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:24.623612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-074d6c35-7945-4152-b964-1c2f9bc42eb0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:24.624389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:24.625337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 78/149] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:37:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:24.626349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074d6c35-7945-4152-b964-1c2f9bc42eb0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:24.636155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 72/150] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:24.641625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:24.646046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:24.727324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074d6c35-7945-4152-b964-1c2f9bc42eb0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:24.746214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246e78f3-317c-4a1d-93b6-2571f8eb58c9 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:24.761618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d8a32b-6884-4dbe-a047-510e2f711855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:24.762339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d8a32b-6884-4dbe-a047-510e2f711855 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.762622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d8a32b-6884-4dbe-a047-510e2f711855 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.762900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d8a32b-6884-4dbe-a047-510e2f711855 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.763993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d8a32b-6884-4dbe-a047-510e2f711855 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:24.765484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d8a32b-6884-4dbe-a047-510e2f711855 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:24.767219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 79/151] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:24.767807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d8a32b-6884-4dbe-a047-510e2f711855 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:24.782896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:24.783517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.783787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.784058 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.784749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:24.785055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:24.786468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:24.795101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 73/152] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:37:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:24.795898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:24.831174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:24.863469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:24.870042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d8a32b-6884-4dbe-a047-510e2f711855 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:24.898808 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a157c-2aa2-4aa4-8614-a0c2584d134e None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:24.904260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c36509-dcf4-4f5e-bd65-e5edb5a7cc71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:24.904921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c36509-dcf4-4f5e-bd65-e5edb5a7cc71 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.905166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c36509-dcf4-4f5e-bd65-e5edb5a7cc71 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.905430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c36509-dcf4-4f5e-bd65-e5edb5a7cc71 None admin] PATH_INFO: `/v3/users/f8293432550d4b3082d556e9c2aae2ac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.918395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c36509-dcf4-4f5e-bd65-e5edb5a7cc71 None admin] RBAC: Authorizing `identity:get_user(user_id=f8293432550d4b3082d556e9c2aae2ac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:24.919841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c36509-dcf4-4f5e-bd65-e5edb5a7cc71 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:24.921543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 80/153] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:24 2022] GET /identity/v3/users/f8293432550d4b3082d556e9c2aae2ac => generated 283 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:24.925081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c36509-dcf4-4f5e-bd65-e5edb5a7cc71 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:24.933490 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3859dd4-3271-4967-9a74-d6ba6565a66c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:24.934228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3859dd4-3271-4967-9a74-d6ba6565a66c None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:24.934513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3859dd4-3271-4967-9a74-d6ba6565a66c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:24.934791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3859dd4-3271-4967-9a74-d6ba6565a66c None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:24.935445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3859dd4-3271-4967-9a74-d6ba6565a66c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:24.936929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3859dd4-3271-4967-9a74-d6ba6565a66c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:24.959093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 74/154] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:24 2022] POST /identity/v3/users => generated 297 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:24.959093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3859dd4-3271-4967-9a74-d6ba6565a66c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:25.045115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c36509-dcf4-4f5e-bd65-e5edb5a7cc71 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:25.073230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:25.082594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3859dd4-3271-4967-9a74-d6ba6565a66c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:25.088842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0ddb26-382d-455e-a4a6-48452a2f99dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:25.089500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0ddb26-382d-455e-a4a6-48452a2f99dc None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:25.089780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0ddb26-382d-455e-a4a6-48452a2f99dc None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:25.090052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0ddb26-382d-455e-a4a6-48452a2f99dc None admin] PATH_INFO: `/v3/users/f8293432550d4b3082d556e9c2aae2ac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:25.091349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0ddb26-382d-455e-a4a6-48452a2f99dc None admin] RBAC: Authorizing `identity:get_user(user_id=f8293432550d4b3082d556e9c2aae2ac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:25.092884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0ddb26-382d-455e-a4a6-48452a2f99dc None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:25.094819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 81/155] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:24 2022] GET /identity/v3/users/f8293432550d4b3082d556e9c2aae2ac => generated 283 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:25.100431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0ddb26-382d-455e-a4a6-48452a2f99dc None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:25.110837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:25.125779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f82e60-641c-454c-a9f7-05e15326174f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:25.126990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f82e60-641c-454c-a9f7-05e15326174f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:25.127279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f82e60-641c-454c-a9f7-05e15326174f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:25.127559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f82e60-641c-454c-a9f7-05e15326174f None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:25.128985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f82e60-641c-454c-a9f7-05e15326174f None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:25.130464 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f82e60-641c-454c-a9f7-05e15326174f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:25.132350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 75/156] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:24 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:25.138664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f82e60-641c-454c-a9f7-05e15326174f None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:25.227561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0ddb26-382d-455e-a4a6-48452a2f99dc None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:25.263251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f82e60-641c-454c-a9f7-05e15326174f None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:25.270492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da9c5b3-7f70-4061-bc32-ca70d87052de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:25.271194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9c5b3-7f70-4061-bc32-ca70d87052de None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:25.271487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9c5b3-7f70-4061-bc32-ca70d87052de None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:25.271749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9c5b3-7f70-4061-bc32-ca70d87052de None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:25.272969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da9c5b3-7f70-4061-bc32-ca70d87052de None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:25.274412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da9c5b3-7f70-4061-bc32-ca70d87052de None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:25.276977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 82/157] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:25 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:25.282757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da9c5b3-7f70-4061-bc32-ca70d87052de None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:25.303117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ad42f9-8d83-4da5-9b8d-d15e40c6ec33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:25.303841 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad42f9-8d83-4da5-9b8d-d15e40c6ec33 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:25.304126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad42f9-8d83-4da5-9b8d-d15e40c6ec33 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:25.304496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad42f9-8d83-4da5-9b8d-d15e40c6ec33 None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545/users/f8293432550d4b3082d556e9c2aae2ac/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:25.306494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ad42f9-8d83-4da5-9b8d-d15e40c6ec33 None admin] RBAC: Authorizing `identity:create_grant(project_id=dc9368fb31aa4024a959720dc9541545, user_id=f8293432550d4b3082d556e9c2aae2ac, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:25.307958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ad42f9-8d83-4da5-9b8d-d15e40c6ec33 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:25.328391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 76/158] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:25 2022] PUT /identity/v3/projects/dc9368fb31aa4024a959720dc9541545/users/f8293432550d4b3082d556e9c2aae2ac/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:37:25.357097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:25.407999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da9c5b3-7f70-4061-bc32-ca70d87052de None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:25.451298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4a82e8-12df-4970-9ba9-8fce5ccc4697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:25.451976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a82e8-12df-4970-9ba9-8fce5ccc4697 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:25.452296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a82e8-12df-4970-9ba9-8fce5ccc4697 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:25.452571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a82e8-12df-4970-9ba9-8fce5ccc4697 None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545/users/f8293432550d4b3082d556e9c2aae2ac/roles/9b956c43ef364a70bf8263ba4314dce7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:25.454777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4a82e8-12df-4970-9ba9-8fce5ccc4697 None admin] RBAC: Authorizing `identity:create_grant(project_id=dc9368fb31aa4024a959720dc9541545, user_id=f8293432550d4b3082d556e9c2aae2ac, role_id=9b956c43ef364a70bf8263ba4314dce7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:25.456274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4a82e8-12df-4970-9ba9-8fce5ccc4697 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:25.473550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 83/159] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:25 2022] PUT /identity/v3/projects/dc9368fb31aa4024a959720dc9541545/users/f8293432550d4b3082d556e9c2aae2ac/roles/9b956c43ef364a70bf8263ba4314dce7 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:27.379379 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5af5c-e93a-44ea-95fa-ea0b084ef527 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.380477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5af5c-e93a-44ea-95fa-ea0b084ef527 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.380982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5af5c-e93a-44ea-95fa-ea0b084ef527 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.382569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 77/160] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:27.388370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0d56f-7430-4a8a-9c21-9bf00aff0298 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.389257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0d56f-7430-4a8a-9c21-9bf00aff0298 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.389728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0d56f-7430-4a8a-9c21-9bf00aff0298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.409284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-1fc0d56f-7430-4a8a-9c21-9bf00aff0298 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:27.410794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1fc0d56f-7430-4a8a-9c21-9bf00aff0298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:27.469950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf86925c-cab6-4feb-91c3-24bdcffeae17 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.470578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf86925c-cab6-4feb-91c3-24bdcffeae17 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.471101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf86925c-cab6-4feb-91c3-24bdcffeae17 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.473016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 78/161] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:27.479270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da31816-ba4b-48b5-b5ab-643d1e00194e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.479982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da31816-ba4b-48b5-b5ab-643d1e00194e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.480567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da31816-ba4b-48b5-b5ab-643d1e00194e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.484828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 84/162] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:27.504401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3da31816-ba4b-48b5-b5ab-643d1e00194e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:27.506021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3da31816-ba4b-48b5-b5ab-643d1e00194e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:27.560048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:27.583542 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 79/163] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:27.625415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc47af-60e0-4f70-bb26-398a8dfe022e None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.625699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc47af-60e0-4f70-bb26-398a8dfe022e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.626203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc47af-60e0-4f70-bb26-398a8dfe022e None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.627764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 85/164] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:27.633467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e7724-36ac-47fa-8ca4-d443ed7c4c99 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.633696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e7724-36ac-47fa-8ca4-d443ed7c4c99 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.633967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e7724-36ac-47fa-8ca4-d443ed7c4c99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.654320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5a6e7724-36ac-47fa-8ca4-d443ed7c4c99 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:27.655537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5a6e7724-36ac-47fa-8ca4-d443ed7c4c99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:27.729960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 80/165] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:27.783285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc47af-60e0-4f70-bb26-398a8dfe022e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:27.784374 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30953b30-6043-46ff-8184-9ee487813555 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.784664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30953b30-6043-46ff-8184-9ee487813555 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.784941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30953b30-6043-46ff-8184-9ee487813555 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.786300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 81/166] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:37:27.792342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ac63c-5892-408a-9531-d0003b01f7bb None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.792615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ac63c-5892-408a-9531-d0003b01f7bb None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.792889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ac63c-5892-408a-9531-d0003b01f7bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.810977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e85ac63c-5892-408a-9531-d0003b01f7bb None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:27.812197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e85ac63c-5892-408a-9531-d0003b01f7bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:27.848303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:27.905057 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 82/167] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:27.906238 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85ac63c-5892-408a-9531-d0003b01f7bb None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:27.908847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc47af-60e0-4f70-bb26-398a8dfe022e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:27.955609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58ef830-97c4-4136-8f49-cbf1c45b30a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:27.956707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58ef830-97c4-4136-8f49-cbf1c45b30a6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:27.957190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58ef830-97c4-4136-8f49-cbf1c45b30a6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:27.957706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58ef830-97c4-4136-8f49-cbf1c45b30a6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:27.961264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:27.970573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58ef830-97c4-4136-8f49-cbf1c45b30a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-c58ef830-97c4-4136-8f49-cbf1c45b30a6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:27.981865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:27.982911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:27.983727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:27.991972 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 86/168] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:27.999443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58ef830-97c4-4136-8f49-cbf1c45b30a6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.038326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85ac63c-5892-408a-9531-d0003b01f7bb None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.075956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dff303-c9b4-4dfc-83fb-8667f30a3512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.076383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dff303-c9b4-4dfc-83fb-8667f30a3512 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.076648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dff303-c9b4-4dfc-83fb-8667f30a3512 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.076919 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dff303-c9b4-4dfc-83fb-8667f30a3512 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.078021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69dff303-c9b4-4dfc-83fb-8667f30a3512 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.079467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69dff303-c9b4-4dfc-83fb-8667f30a3512 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.081224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 83/169] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:28.081820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dff303-c9b4-4dfc-83fb-8667f30a3512 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.112847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58ef830-97c4-4136-8f49-cbf1c45b30a6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.123024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:28.148897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.149511 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.149793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.150109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.150853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] RBAC: Adding query filter params (name=member) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:28.151162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.152570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.162310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 87/170] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:28.163037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.194038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dff303-c9b4-4dfc-83fb-8667f30a3512 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.203888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:28.229771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7295339a-9e6c-40e1-a3b3-4e33b2fa450b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.230401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7295339a-9e6c-40e1-a3b3-4e33b2fa450b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.230688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7295339a-9e6c-40e1-a3b3-4e33b2fa450b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.230965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7295339a-9e6c-40e1-a3b3-4e33b2fa450b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.240951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7295339a-9e6c-40e1-a3b3-4e33b2fa450b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-7295339a-9e6c-40e1-a3b3-4e33b2fa450b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:28.246759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:28.247840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 84/171] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:28.249048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7295339a-9e6c-40e1-a3b3-4e33b2fa450b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.270870 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c23894-32c5-48df-a199-aa1bcd8c395c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.311916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66440a2-1e6a-4c2a-93d3-a00c5e4d15e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.312595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66440a2-1e6a-4c2a-93d3-a00c5e4d15e1 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.312881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66440a2-1e6a-4c2a-93d3-a00c5e4d15e1 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.313159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66440a2-1e6a-4c2a-93d3-a00c5e4d15e1 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.313820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66440a2-1e6a-4c2a-93d3-a00c5e4d15e1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.315306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66440a2-1e6a-4c2a-93d3-a00c5e4d15e1 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.352164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 88/172] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:28 2022] POST /identity/v3/users => generated 297 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:28.352164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66440a2-1e6a-4c2a-93d3-a00c5e4d15e1 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.363215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7295339a-9e6c-40e1-a3b3-4e33b2fa450b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.374264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:28.405018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f162ec-fcc4-4e03-97f1-9bc8e9bfca91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.405794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f162ec-fcc4-4e03-97f1-9bc8e9bfca91 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.406152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f162ec-fcc4-4e03-97f1-9bc8e9bfca91 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.406484 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f162ec-fcc4-4e03-97f1-9bc8e9bfca91 None admin] PATH_INFO: `/v3/users/aa53747453b4458fa92266c9f97feb96` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.423254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f162ec-fcc4-4e03-97f1-9bc8e9bfca91 None admin] RBAC: Authorizing `identity:get_user(user_id=aa53747453b4458fa92266c9f97feb96)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.425185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f162ec-fcc4-4e03-97f1-9bc8e9bfca91 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.426665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 85/173] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/users/aa53747453b4458fa92266c9f97feb96 => generated 297 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:28.427302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f162ec-fcc4-4e03-97f1-9bc8e9bfca91 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.461547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:28.468889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66440a2-1e6a-4c2a-93d3-a00c5e4d15e1 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.511492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241035ac-3b7b-431b-b73a-5772ad565584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.512165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241035ac-3b7b-431b-b73a-5772ad565584 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.512474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241035ac-3b7b-431b-b73a-5772ad565584 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.512746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241035ac-3b7b-431b-b73a-5772ad565584 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.524402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241035ac-3b7b-431b-b73a-5772ad565584 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-241035ac-3b7b-431b-b73a-5772ad565584 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:28.529836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:28.530737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:28.531583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 89/174] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:28.533203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241035ac-3b7b-431b-b73a-5772ad565584 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.557523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f162ec-fcc4-4e03-97f1-9bc8e9bfca91 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.602947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.604823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.605256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.605709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.606580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:28.607077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.608516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.619991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 86/175] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:28.620972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.637119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:28.655611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241035ac-3b7b-431b-b73a-5772ad565584 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.696574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f262e74c-a08d-4c85-bb2f-46a968e2a4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.697185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f262e74c-a08d-4c85-bb2f-46a968e2a4ab None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.697465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f262e74c-a08d-4c85-bb2f-46a968e2a4ab None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.697764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f262e74c-a08d-4c85-bb2f-46a968e2a4ab None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.699749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f262e74c-a08d-4c85-bb2f-46a968e2a4ab None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.701241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f262e74c-a08d-4c85-bb2f-46a968e2a4ab None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.703205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 90/176] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:28.703990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f262e74c-a08d-4c85-bb2f-46a968e2a4ab None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.727397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:28.747046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6f01ae-6fff-4c13-8fb1-0562f9503fa2 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.789924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.790604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.790952 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.791226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.792045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:28.792420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.793863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.803006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 87/177] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:28.809013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.843038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f262e74c-a08d-4c85-bb2f-46a968e2a4ab None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.891613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630dab0-8dde-437b-9a86-4eeda5c25dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.893143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630dab0-8dde-437b-9a86-4eeda5c25dc8 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.893634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630dab0-8dde-437b-9a86-4eeda5c25dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.894522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630dab0-8dde-437b-9a86-4eeda5c25dc8 None admin] PATH_INFO: `/v3/users/f8293432550d4b3082d556e9c2aae2ac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.896287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6630dab0-8dde-437b-9a86-4eeda5c25dc8 None admin] RBAC: Authorizing `identity:get_user(user_id=f8293432550d4b3082d556e9c2aae2ac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.898924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6630dab0-8dde-437b-9a86-4eeda5c25dc8 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.902392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 91/178] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/users/f8293432550d4b3082d556e9c2aae2ac => generated 283 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:28.903449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630dab0-8dde-437b-9a86-4eeda5c25dc8 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:28.919073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e3d1eb-0fc1-4c3a-9b04-011ca151c799 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:28.944022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:28.957920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d64d3e0-d179-4b9c-a5df-ac52d38ea75e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:28.958507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d64d3e0-d179-4b9c-a5df-ac52d38ea75e None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:28.958788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d64d3e0-d179-4b9c-a5df-ac52d38ea75e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:28.959064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d64d3e0-d179-4b9c-a5df-ac52d38ea75e None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545/users/aa53747453b4458fa92266c9f97feb96/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:28.961221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d64d3e0-d179-4b9c-a5df-ac52d38ea75e None admin] RBAC: Authorizing `identity:create_grant(project_id=dc9368fb31aa4024a959720dc9541545, user_id=aa53747453b4458fa92266c9f97feb96, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:28.963200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d64d3e0-d179-4b9c-a5df-ac52d38ea75e None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:28.988361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 88/179] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:28 2022] PUT /identity/v3/projects/dc9368fb31aa4024a959720dc9541545/users/aa53747453b4458fa92266c9f97feb96/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:28.991898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d64d3e0-d179-4b9c-a5df-ac52d38ea75e None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:29.018416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:29.052121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630dab0-8dde-437b-9a86-4eeda5c25dc8 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:29.097584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0db6c2e-2bd1-436c-a015-2ffb3cfbfacf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:29.098246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db6c2e-2bd1-436c-a015-2ffb3cfbfacf None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:29.098527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db6c2e-2bd1-436c-a015-2ffb3cfbfacf None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:29.098801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db6c2e-2bd1-436c-a015-2ffb3cfbfacf None admin] PATH_INFO: `/v3/users/f8293432550d4b3082d556e9c2aae2ac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:29.100109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0db6c2e-2bd1-436c-a015-2ffb3cfbfacf None admin] RBAC: Authorizing `identity:get_user(user_id=f8293432550d4b3082d556e9c2aae2ac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:29.101651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0db6c2e-2bd1-436c-a015-2ffb3cfbfacf None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:29.105087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 92/180] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/users/f8293432550d4b3082d556e9c2aae2ac => generated 283 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:29.111892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0db6c2e-2bd1-436c-a015-2ffb3cfbfacf None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:29.222110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d64d3e0-d179-4b9c-a5df-ac52d38ea75e None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:29.267068 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f7eebc-e32a-44e2-97d6-3f783915a7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:29.267685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f7eebc-e32a-44e2-97d6-3f783915a7d3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:29.267970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f7eebc-e32a-44e2-97d6-3f783915a7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:29.268849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f7eebc-e32a-44e2-97d6-3f783915a7d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:29.271262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:29.283180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0db6c2e-2bd1-436c-a015-2ffb3cfbfacf None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:29.320989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3702d9-f36e-4873-ae41-68ce26ffbcf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:29.321753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3702d9-f36e-4873-ae41-68ce26ffbcf9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:29.322640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3702d9-f36e-4873-ae41-68ce26ffbcf9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:29.322983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3702d9-f36e-4873-ae41-68ce26ffbcf9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:29.364597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f7eebc-e32a-44e2-97d6-3f783915a7d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:29.366091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f7eebc-e32a-44e2-97d6-3f783915a7d3 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:29.387943 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:29.424879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3702d9-f36e-4873-ae41-68ce26ffbcf9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:29.426397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3702d9-f36e-4873-ae41-68ce26ffbcf9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:29.545757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 89/181] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 557 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:29.552207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f7eebc-e32a-44e2-97d6-3f783915a7d3 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:29.624378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:29.641860 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 93/182] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 531 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 24 08:37:29.647975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3702d9-f36e-4873-ae41-68ce26ffbcf9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:29.762346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f7eebc-e32a-44e2-97d6-3f783915a7d3 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:29.781689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:29.829565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5901cc2-69d6-41c2-9b43-0c37bca56934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:29.830136 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5901cc2-69d6-41c2-9b43-0c37bca56934 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:29.830410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5901cc2-69d6-41c2-9b43-0c37bca56934 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:29.830677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5901cc2-69d6-41c2-9b43-0c37bca56934 None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:29.832438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5901cc2-69d6-41c2-9b43-0c37bca56934 None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:29.833920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5901cc2-69d6-41c2-9b43-0c37bca56934 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:29.840479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 90/183] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:29 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:29.846782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5901cc2-69d6-41c2-9b43-0c37bca56934 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:29.863677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3702d9-f36e-4873-ae41-68ce26ffbcf9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:29.922868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf19162e-fdf3-46c3-816e-414809e8f8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:29.923778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf19162e-fdf3-46c3-816e-414809e8f8a0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:29.924053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf19162e-fdf3-46c3-816e-414809e8f8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:29.924398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf19162e-fdf3-46c3-816e-414809e8f8a0 None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:29.926002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf19162e-fdf3-46c3-816e-414809e8f8a0 None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:29.927739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf19162e-fdf3-46c3-816e-414809e8f8a0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:29.929533 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 94/184] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:29 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:29.936619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf19162e-fdf3-46c3-816e-414809e8f8a0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:30.008144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:30.056710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5901cc2-69d6-41c2-9b43-0c37bca56934 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:30.066307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf19162e-fdf3-46c3-816e-414809e8f8a0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:30.091988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc18558-0712-431a-ac9e-dc5bb6a8435b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:30.092909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc18558-0712-431a-ac9e-dc5bb6a8435b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:30.093361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc18558-0712-431a-ac9e-dc5bb6a8435b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:30.093826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc18558-0712-431a-ac9e-dc5bb6a8435b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:30.094718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc18558-0712-431a-ac9e-dc5bb6a8435b None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=dc9368fb31aa4024a959720dc9541545, user.id=f8293432550d4b3082d556e9c2aae2ac) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:30.095200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc18558-0712-431a-ac9e-dc5bb6a8435b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:30.096837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc18558-0712-431a-ac9e-dc5bb6a8435b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:30.103713 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 91/185] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:29 2022] GET /identity/v3/role_assignments?user.id=f8293432550d4b3082d556e9c2aae2ac&scope.project.id=dc9368fb31aa4024a959720dc9541545&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 608 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:30.120365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31e9f2d-1eb6-4300-8258-fb460e27cd70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:30.120994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31e9f2d-1eb6-4300-8258-fb460e27cd70 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:30.121269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31e9f2d-1eb6-4300-8258-fb460e27cd70 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:30.121537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31e9f2d-1eb6-4300-8258-fb460e27cd70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:30.122270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31e9f2d-1eb6-4300-8258-fb460e27cd70 None admin] RBAC: Adding query filter params (role.id=9b956c43ef364a70bf8263ba4314dce7, scope.project.id=dc9368fb31aa4024a959720dc9541545, user.id=f8293432550d4b3082d556e9c2aae2ac) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:30.122577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31e9f2d-1eb6-4300-8258-fb460e27cd70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:30.124017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31e9f2d-1eb6-4300-8258-fb460e27cd70 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:30.131161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 95/186] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:29 2022] GET /identity/v3/role_assignments?user.id=f8293432550d4b3082d556e9c2aae2ac&scope.project.id=dc9368fb31aa4024a959720dc9541545&role.id=9b956c43ef364a70bf8263ba4314dce7 => generated 608 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:30.933556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7feb2f4-8d8f-4990-97be-e0fb348d8ed5 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:30.934077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7feb2f4-8d8f-4990-97be-e0fb348d8ed5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:30.934077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7feb2f4-8d8f-4990-97be-e0fb348d8ed5 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:30.935572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 92/187] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:30.941380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6472f0c7-b48c-4aa0-9e6e-cd9f7c7b963b None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:30.941878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6472f0c7-b48c-4aa0-9e6e-cd9f7c7b963b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:30.941958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6472f0c7-b48c-4aa0-9e6e-cd9f7c7b963b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:30.960507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6472f0c7-b48c-4aa0-9e6e-cd9f7c7b963b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:30.961694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6472f0c7-b48c-4aa0-9e6e-cd9f7c7b963b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:31.038014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fec501-61a3-4957-a491-6ea6cedd8f16 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.038335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fec501-61a3-4957-a491-6ea6cedd8f16 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.038560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fec501-61a3-4957-a491-6ea6cedd8f16 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.040085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 93/188] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:37:31.040220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 96/189] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:31.046131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a9275e-d8cf-49d7-bb05-5db921f65b4d None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.046414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a9275e-d8cf-49d7-bb05-5db921f65b4d None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.046687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a9275e-d8cf-49d7-bb05-5db921f65b4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.064684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-84a9275e-d8cf-49d7-bb05-5db921f65b4d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:31.065858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-84a9275e-d8cf-49d7-bb05-5db921f65b4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:31.105094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:31.139566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 94/190] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:31.318006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6472f0c7-b48c-4aa0-9e6e-cd9f7c7b963b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:31.374900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:31.405872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a9275e-d8cf-49d7-bb05-5db921f65b4d None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:31.429084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6472f0c7-b48c-4aa0-9e6e-cd9f7c7b963b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:31.472598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69df128-b1cb-47f4-a67f-60c986bc41c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:31.473234 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69df128-b1cb-47f4-a67f-60c986bc41c8 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.473515 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69df128-b1cb-47f4-a67f-60c986bc41c8 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.473787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69df128-b1cb-47f4-a67f-60c986bc41c8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.485107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69df128-b1cb-47f4-a67f-60c986bc41c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-c69df128-b1cb-47f4-a67f-60c986bc41c8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:31.490031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:31.490964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:31.492043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 97/191] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:31.492759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541d3292-13ed-4950-9fbb-10595fb91dfe None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.493006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541d3292-13ed-4950-9fbb-10595fb91dfe None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.493271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541d3292-13ed-4950-9fbb-10595fb91dfe None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.494710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 98/192] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:31.497222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541d3292-13ed-4950-9fbb-10595fb91dfe None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:31.513388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a9275e-d8cf-49d7-bb05-5db921f65b4d None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:31.550647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f871bc-0be6-4b66-a027-07be86a7d26f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:31.551306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f871bc-0be6-4b66-a027-07be86a7d26f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.551585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f871bc-0be6-4b66-a027-07be86a7d26f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.551855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f871bc-0be6-4b66-a027-07be86a7d26f None admin] PATH_INFO: `/v3/domains/default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.553087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f871bc-0be6-4b66-a027-07be86a7d26f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:31.554577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f871bc-0be6-4b66-a027-07be86a7d26f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:31.556307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 95/193] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:31.557535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0215a-38c4-4f62-a24e-297b084c8b51 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.557816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0215a-38c4-4f62-a24e-297b084c8b51 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.558088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0215a-38c4-4f62-a24e-297b084c8b51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.577548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-0aa0215a-38c4-4f62-a24e-297b084c8b51 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:31.578945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0aa0215a-38c4-4f62-a24e-297b084c8b51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:31.606655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541d3292-13ed-4950-9fbb-10595fb91dfe None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:31.613987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:31.620948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:31.644722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:31.645328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.645612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.645927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.646586 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:31.646908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:31.648313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:31.648961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 96/194] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:31.653041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa0215a-38c4-4f62-a24e-297b084c8b51 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:31.659202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 99/195] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:31.664502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:31.764604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa0215a-38c4-4f62-a24e-297b084c8b51 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:31.766636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618afc2-ab18-4b44-9a38-77aa89fd12d1 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:31.801389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428141ac-65d7-4c10-b80c-2f6142393cd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:31.801637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43adb75d-6bde-49d6-9e5a-354765bf2954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:31.801995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428141ac-65d7-4c10-b80c-2f6142393cd1 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.802161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43adb75d-6bde-49d6-9e5a-354765bf2954 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.802255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428141ac-65d7-4c10-b80c-2f6142393cd1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.802433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43adb75d-6bde-49d6-9e5a-354765bf2954 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.802583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428141ac-65d7-4c10-b80c-2f6142393cd1 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.802730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43adb75d-6bde-49d6-9e5a-354765bf2954 None admin] PATH_INFO: `/v3/users/8e83a1b2271a4d5484cb49ad7868be7d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.803283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428141ac-65d7-4c10-b80c-2f6142393cd1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:31.804723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428141ac-65d7-4c10-b80c-2f6142393cd1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:31.818016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43adb75d-6bde-49d6-9e5a-354765bf2954 None admin] RBAC: Authorizing `identity:get_user(user_id=8e83a1b2271a4d5484cb49ad7868be7d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:31.819492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43adb75d-6bde-49d6-9e5a-354765bf2954 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:31.821217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 100/196] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:31 2022] GET /identity/v3/users/8e83a1b2271a4d5484cb49ad7868be7d => generated 297 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:31.827470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43adb75d-6bde-49d6-9e5a-354765bf2954 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:31.833077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 97/197] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:31 2022] POST /identity/v3/users => generated 293 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:31.862268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:31.928486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428141ac-65d7-4c10-b80c-2f6142393cd1 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:31.941769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43adb75d-6bde-49d6-9e5a-354765bf2954 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:31.961217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:31.980264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0128d0-071f-4de0-aae4-870756ba6074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:31.980914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0128d0-071f-4de0-aae4-870756ba6074 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:31.981191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0128d0-071f-4de0-aae4-870756ba6074 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:31.981462 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0128d0-071f-4de0-aae4-870756ba6074 None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:31.982626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0128d0-071f-4de0-aae4-870756ba6074 None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:31.984073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0128d0-071f-4de0-aae4-870756ba6074 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:31.985824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 101/198] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:31 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:31.991970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0128d0-071f-4de0-aae4-870756ba6074 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:32.065287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428141ac-65d7-4c10-b80c-2f6142393cd1 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:32.104974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:32.107904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0128d0-071f-4de0-aae4-870756ba6074 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:32.115972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eeff3f-41f4-46e3-86b8-944a8db456fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:32.116921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eeff3f-41f4-46e3-86b8-944a8db456fb None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:32.117210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eeff3f-41f4-46e3-86b8-944a8db456fb None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:32.117480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eeff3f-41f4-46e3-86b8-944a8db456fb None admin] PATH_INFO: `/v3/roles/member` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:32.130281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eeff3f-41f4-46e3-86b8-944a8db456fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-f4eeff3f-41f4-46e3-86b8-944a8db456fb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:32.136878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:32.137818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:32.138618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 98/199] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:37:32.145283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eeff3f-41f4-46e3-86b8-944a8db456fb None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:32.157935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb151913-1bad-4016-82f9-33761774439b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:32.158970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb151913-1bad-4016-82f9-33761774439b None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:32.158970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb151913-1bad-4016-82f9-33761774439b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:32.159230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb151913-1bad-4016-82f9-33761774439b None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545/users/8e83a1b2271a4d5484cb49ad7868be7d/roles/6d01ecb1b0554e219119728a79f21015` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:32.161872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb151913-1bad-4016-82f9-33761774439b None admin] RBAC: Authorizing `identity:create_grant(project_id=dc9368fb31aa4024a959720dc9541545, user_id=8e83a1b2271a4d5484cb49ad7868be7d, role_id=6d01ecb1b0554e219119728a79f21015)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:32.163956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb151913-1bad-4016-82f9-33761774439b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:32.186115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 102/200] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:37:31 2022] PUT /identity/v3/projects/dc9368fb31aa4024a959720dc9541545/users/8e83a1b2271a4d5484cb49ad7868be7d/roles/6d01ecb1b0554e219119728a79f21015 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:37:32.254792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:32.274158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eeff3f-41f4-46e3-86b8-944a8db456fb None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:32.320899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:32.321543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:32.321863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:32.322143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:32.322858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] RBAC: Adding query filter params (name=member) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:32.323167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:32.324816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:32.335397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 99/201] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:32.341568 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb151913-1bad-4016-82f9-33761774439b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:32.450776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb151913-1bad-4016-82f9-33761774439b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:32.487252 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd78778-919d-4f29-9be6-fb7cd56301fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:32.487903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd78778-919d-4f29-9be6-fb7cd56301fa None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:32.488222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd78778-919d-4f29-9be6-fb7cd56301fa None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:32.488503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd78778-919d-4f29-9be6-fb7cd56301fa None admin] PATH_INFO: `/v3/users/aa53747453b4458fa92266c9f97feb96` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:32.489648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd78778-919d-4f29-9be6-fb7cd56301fa None admin] RBAC: Authorizing `identity:get_user(user_id=aa53747453b4458fa92266c9f97feb96)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:32.491107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd78778-919d-4f29-9be6-fb7cd56301fa None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:32.493366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 103/202] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:32 2022] GET /identity/v3/users/aa53747453b4458fa92266c9f97feb96 => generated 297 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:32.499277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:32.608235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea5673-4d8a-4078-b427-d46761a0262f None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:32.645129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29955869-fccf-4948-bdd4-40858f95faed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:32.646126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29955869-fccf-4948-bdd4-40858f95faed None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:32.646606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29955869-fccf-4948-bdd4-40858f95faed None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:32.647096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29955869-fccf-4948-bdd4-40858f95faed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:32.666557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:32.725881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29955869-fccf-4948-bdd4-40858f95faed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:32.727333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29955869-fccf-4948-bdd4-40858f95faed None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:32.835604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 100/203] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:32.841564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd78778-919d-4f29-9be6-fb7cd56301fa None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:32.871335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:32.954161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd78778-919d-4f29-9be6-fb7cd56301fa None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:32.995942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc1ff6a-7a89-489f-9091-1a81b0a6b99e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:32.996636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc1ff6a-7a89-489f-9091-1a81b0a6b99e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:32.996914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc1ff6a-7a89-489f-9091-1a81b0a6b99e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:32.997181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc1ff6a-7a89-489f-9091-1a81b0a6b99e None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:32.998435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc1ff6a-7a89-489f-9091-1a81b0a6b99e None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:32.999925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc1ff6a-7a89-489f-9091-1a81b0a6b99e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:33.001909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 104/204] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:32 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:33.007783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29955869-fccf-4948-bdd4-40858f95faed None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:33.069907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:33.130293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29955869-fccf-4948-bdd4-40858f95faed None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:33.169699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11c66ea-49b9-4bc0-8f0a-75cebb0790d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:33.170322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11c66ea-49b9-4bc0-8f0a-75cebb0790d7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:33.170654 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11c66ea-49b9-4bc0-8f0a-75cebb0790d7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:33.170936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11c66ea-49b9-4bc0-8f0a-75cebb0790d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:33.171685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11c66ea-49b9-4bc0-8f0a-75cebb0790d7 None admin] RBAC: Adding query filter params (role.id=05482458bad348e0be384b75bfd31eac, scope.project.id=dc9368fb31aa4024a959720dc9541545, user.id=aa53747453b4458fa92266c9f97feb96) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:33.171994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11c66ea-49b9-4bc0-8f0a-75cebb0790d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:33.173539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11c66ea-49b9-4bc0-8f0a-75cebb0790d7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:33.180777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 101/205] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:33 2022] GET /identity/v3/role_assignments?user.id=aa53747453b4458fa92266c9f97feb96&scope.project.id=dc9368fb31aa4024a959720dc9541545&role.id=05482458bad348e0be384b75bfd31eac => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:34.212047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b724cdcf-40e4-49ea-924b-bd83b5ccdb72 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.213097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b724cdcf-40e4-49ea-924b-bd83b5ccdb72 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.213376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b724cdcf-40e4-49ea-924b-bd83b5ccdb72 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.215767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 105/206] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:34.221249 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ade746-224e-4101-9c37-5ad18de426ad None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.221679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ade746-224e-4101-9c37-5ad18de426ad None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.221902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ade746-224e-4101-9c37-5ad18de426ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.242158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-98ade746-224e-4101-9c37-5ad18de426ad None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:34.243312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-98ade746-224e-4101-9c37-5ad18de426ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:34.317124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 102/207] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:34.363924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314904c5-5e56-47e1-83aa-b6d22e93d110 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.364272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314904c5-5e56-47e1-83aa-b6d22e93d110 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.364501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314904c5-5e56-47e1-83aa-b6d22e93d110 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.365978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 106/208] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:34.371521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d2995-7390-401a-9466-fa910098bbfd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.371807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d2995-7390-401a-9466-fa910098bbfd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.372078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d2995-7390-401a-9466-fa910098bbfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.392070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c88d2995-7390-401a-9466-fa910098bbfd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:34.393324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c88d2995-7390-401a-9466-fa910098bbfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:34.433030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:34.467563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 103/209] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:34.506528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc825fd-3f99-4491-be0b-d71b10b651a3 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.506805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc825fd-3f99-4491-be0b-d71b10b651a3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.507004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc825fd-3f99-4491-be0b-d71b10b651a3 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.508518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 107/210] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:34.513706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb0e3fe-4e6a-4f28-b32d-4f77d98030e4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.513994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb0e3fe-4e6a-4f28-b32d-4f77d98030e4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.514262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb0e3fe-4e6a-4f28-b32d-4f77d98030e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.532394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2fb0e3fe-4e6a-4f28-b32d-4f77d98030e4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:34.533573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2fb0e3fe-4e6a-4f28-b32d-4f77d98030e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:34.591167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc825fd-3f99-4491-be0b-d71b10b651a3 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:34.599694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 104/211] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:34.686694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:34.693479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc825fd-3f99-4491-be0b-d71b10b651a3 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:34.728407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d74cbc-b13d-4f5c-b9fe-7c0fbe3a6907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:34.729045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d74cbc-b13d-4f5c-b9fe-7c0fbe3a6907 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.729322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d74cbc-b13d-4f5c-b9fe-7c0fbe3a6907 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.729592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d74cbc-b13d-4f5c-b9fe-7c0fbe3a6907 None admin] PATH_INFO: `/v3/roles/member` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.739998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d74cbc-b13d-4f5c-b9fe-7c0fbe3a6907 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-50d74cbc-b13d-4f5c-b9fe-7c0fbe3a6907 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:34.745822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:34.746859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:34.747638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 108/212] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:37:34.753259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb0e3fe-4e6a-4f28-b32d-4f77d98030e4 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:34.761111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d74cbc-b13d-4f5c-b9fe-7c0fbe3a6907 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:34.865555 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb0e3fe-4e6a-4f28-b32d-4f77d98030e4 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:34.867902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:34.879733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d74cbc-b13d-4f5c-b9fe-7c0fbe3a6907 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:34.903129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:34.903734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.904012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.904320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.904986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] RBAC: Adding query filter params (name=member) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:34.905291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:34.906663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:34.913425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c4b936-7b25-46c2-9344-28551dda9680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:34.914046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c4b936-7b25-46c2-9344-28551dda9680 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:34.914317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c4b936-7b25-46c2-9344-28551dda9680 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:34.914584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c4b936-7b25-46c2-9344-28551dda9680 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:34.915415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 105/213] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:34.915711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c4b936-7b25-46c2-9344-28551dda9680 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:34.916386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:34.917522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c4b936-7b25-46c2-9344-28551dda9680 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:34.919108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 109/214] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:34.922194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c4b936-7b25-46c2-9344-28551dda9680 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:34.949243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:35.024044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12655308-a849-4ccb-b4d4-7739da924833 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:35.027295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c4b936-7b25-46c2-9344-28551dda9680 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:35.058245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7057f2ad-b2e3-4cba-b044-647fc435223b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:35.058878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7057f2ad-b2e3-4cba-b044-647fc435223b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:35.059157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7057f2ad-b2e3-4cba-b044-647fc435223b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:35.059428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7057f2ad-b2e3-4cba-b044-647fc435223b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:35.062163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d0f30-4c5d-4fe5-a7d5-333b5cfb5b99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:35.062786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d0f30-4c5d-4fe5-a7d5-333b5cfb5b99 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:35.063063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d0f30-4c5d-4fe5-a7d5-333b5cfb5b99 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:35.063333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d0f30-4c5d-4fe5-a7d5-333b5cfb5b99 None admin] PATH_INFO: `/v3/users/ab6683aa590e4d15b0e747b0707c2ee2` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:35.069702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7057f2ad-b2e3-4cba-b044-647fc435223b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-7057f2ad-b2e3-4cba-b044-647fc435223b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:35.075652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:35.076600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 106/215] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:35.077602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7057f2ad-b2e3-4cba-b044-647fc435223b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:35.079899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816d0f30-4c5d-4fe5-a7d5-333b5cfb5b99 None admin] RBAC: Authorizing `identity:get_user(user_id=ab6683aa590e4d15b0e747b0707c2ee2)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:35.081396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816d0f30-4c5d-4fe5-a7d5-333b5cfb5b99 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:35.083181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 110/216] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:34 2022] GET /identity/v3/users/ab6683aa590e4d15b0e747b0707c2ee2 => generated 293 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:35.083820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d0f30-4c5d-4fe5-a7d5-333b5cfb5b99 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:35.105543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:35.168368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:35.179063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7057f2ad-b2e3-4cba-b044-647fc435223b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:35.187077 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d0f30-4c5d-4fe5-a7d5-333b5cfb5b99 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:35.211821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cdb949-1cde-45f5-9629-dd390680b598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:35.212446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cdb949-1cde-45f5-9629-dd390680b598 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:35.212760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cdb949-1cde-45f5-9629-dd390680b598 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:35.213025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cdb949-1cde-45f5-9629-dd390680b598 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:35.213755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cdb949-1cde-45f5-9629-dd390680b598 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:35.215179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cdb949-1cde-45f5-9629-dd390680b598 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:35.219406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:35.220005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:35.220319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:35.220593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:35.221252 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] RBAC: Adding query filter params (name=reader) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:35.221555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:35.222915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:35.236587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 111/217] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:35 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:35.236587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:35.239292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 107/218] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:35 2022] POST /identity/v3/users => generated 293 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:35.240105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cdb949-1cde-45f5-9629-dd390680b598 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:35.333539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:35.356773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cdb949-1cde-45f5-9629-dd390680b598 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:35.360121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28af1882-00da-4ab6-9b42-a79182e97131 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:35.395440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815b4b81-0345-487f-b4e5-a3a6e7358c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:35.396087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815b4b81-0345-487f-b4e5-a3a6e7358c0e None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:35.396394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815b4b81-0345-487f-b4e5-a3a6e7358c0e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:35.396666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815b4b81-0345-487f-b4e5-a3a6e7358c0e None admin] PATH_INFO: `/v3/users/8e83a1b2271a4d5484cb49ad7868be7d` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:35.398195 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815b4b81-0345-487f-b4e5-a3a6e7358c0e None admin] RBAC: Authorizing `identity:get_user(user_id=8e83a1b2271a4d5484cb49ad7868be7d)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:35.399802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815b4b81-0345-487f-b4e5-a3a6e7358c0e None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:35.402072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 108/219] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:35 2022] GET /identity/v3/users/8e83a1b2271a4d5484cb49ad7868be7d => generated 297 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:35.407984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815b4b81-0345-487f-b4e5-a3a6e7358c0e None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:35.408574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:35.409210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:35.409492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:35.409763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:35.410496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] RBAC: Adding query filter params (role.id=05482458bad348e0be384b75bfd31eac, user.id=ab6683aa590e4d15b0e747b0707c2ee2, scope.system=all) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:35.411173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:35.413715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:35.421280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 112/220] 10.222.0.39 () {56 vars in 1308 bytes} [Mon Oct 24 08:37:35 2022] GET /identity/v3/role_assignments?user.id=ab6683aa590e4d15b0e747b0707c2ee2&scope.system=all&role.id=05482458bad348e0be384b75bfd31eac => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:35.537062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815b4b81-0345-487f-b4e5-a3a6e7358c0e None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:35.579622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ca6ff5-802e-4fa5-83b6-1a121e8b569a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:35.580294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca6ff5-802e-4fa5-83b6-1a121e8b569a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:35.580581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca6ff5-802e-4fa5-83b6-1a121e8b569a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:35.580857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca6ff5-802e-4fa5-83b6-1a121e8b569a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:35.601624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:35.658706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ca6ff5-802e-4fa5-83b6-1a121e8b569a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:35.660576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ca6ff5-802e-4fa5-83b6-1a121e8b569a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:35.773382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 109/221] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 24 08:37:35.781259 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:35.794162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:35.892796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a72fd3-34e8-447a-bbb8-1e5f9dc00b39 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:35.929060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e18bf0-8f9a-4e81-bc7e-159a7dced346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:35.929693 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e18bf0-8f9a-4e81-bc7e-159a7dced346 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:35.929974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e18bf0-8f9a-4e81-bc7e-159a7dced346 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:35.930246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e18bf0-8f9a-4e81-bc7e-159a7dced346 None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:35.931478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e18bf0-8f9a-4e81-bc7e-159a7dced346 None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:35.932948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e18bf0-8f9a-4e81-bc7e-159a7dced346 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:35.934622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 113/222] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:35 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:35.940569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ca6ff5-802e-4fa5-83b6-1a121e8b569a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:36.005607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:36.053624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ca6ff5-802e-4fa5-83b6-1a121e8b569a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:36.089900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d568ccf5-e313-48da-9731-697f1671be90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:36.090887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d568ccf5-e313-48da-9731-697f1671be90 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:36.091365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d568ccf5-e313-48da-9731-697f1671be90 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:36.091832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d568ccf5-e313-48da-9731-697f1671be90 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:36.092778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d568ccf5-e313-48da-9731-697f1671be90 None admin] RBAC: Adding query filter params (role.id=6d01ecb1b0554e219119728a79f21015, scope.project.id=dc9368fb31aa4024a959720dc9541545, user.id=8e83a1b2271a4d5484cb49ad7868be7d) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:36.093286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d568ccf5-e313-48da-9731-697f1671be90 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:36.094929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d568ccf5-e313-48da-9731-697f1671be90 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:36.101457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 110/223] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:37:35 2022] GET /identity/v3/role_assignments?user.id=8e83a1b2271a4d5484cb49ad7868be7d&scope.project.id=dc9368fb31aa4024a959720dc9541545&role.id=6d01ecb1b0554e219119728a79f21015 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:37.675678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ae3c51-25ec-4155-b02c-aecdde4d3946 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:37.676450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ae3c51-25ec-4155-b02c-aecdde4d3946 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:37.676450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ae3c51-25ec-4155-b02c-aecdde4d3946 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:37.677741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 114/224] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:37.683443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb317018-c6e8-4273-8e98-c333453e216e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:37.683870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb317018-c6e8-4273-8e98-c333453e216e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:37.684046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb317018-c6e8-4273-8e98-c333453e216e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:37.687923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fbc1d7-a7a6-48d5-8e18-a6c8d135b202 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:37.688234 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fbc1d7-a7a6-48d5-8e18-a6c8d135b202 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:37.688514 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fbc1d7-a7a6-48d5-8e18-a6c8d135b202 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:37.689994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 115/225] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:37:37.695067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c73fa-676a-4cf7-9eea-2cf69f59fbd4 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:37.695348 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c73fa-676a-4cf7-9eea-2cf69f59fbd4 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:37.695616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c73fa-676a-4cf7-9eea-2cf69f59fbd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:37.703251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-bb317018-c6e8-4273-8e98-c333453e216e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:37.704486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bb317018-c6e8-4273-8e98-c333453e216e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:37.714564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c21c73fa-676a-4cf7-9eea-2cf69f59fbd4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:37.715740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c21c73fa-676a-4cf7-9eea-2cf69f59fbd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:37.772508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 111/226] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:37.773751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44749f-7158-4212-9dd3-057b22fc45f1 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:37.774035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44749f-7158-4212-9dd3-057b22fc45f1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:37.774309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44749f-7158-4212-9dd3-057b22fc45f1 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:37.776156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 112/227] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:37.782005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9befbd88-ba37-4d22-b776-11443f72c376 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:37.782383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9befbd88-ba37-4d22-b776-11443f72c376 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:37.782635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9befbd88-ba37-4d22-b776-11443f72c376 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:37.792858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 116/228] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:37.800858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9befbd88-ba37-4d22-b776-11443f72c376 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:37.802034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9befbd88-ba37-4d22-b776-11443f72c376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:37.838555 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:37.872847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 113/229] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:38.048481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21c73fa-676a-4cf7-9eea-2cf69f59fbd4 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.057705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:38.061303 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9befbd88-ba37-4d22-b776-11443f72c376 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.153303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21c73fa-676a-4cf7-9eea-2cf69f59fbd4 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.172463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9befbd88-ba37-4d22-b776-11443f72c376 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.187769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77e52a7-b892-4d42-af06-cf92823b4e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.188429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77e52a7-b892-4d42-af06-cf92823b4e39 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.188718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77e52a7-b892-4d42-af06-cf92823b4e39 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.188990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77e52a7-b892-4d42-af06-cf92823b4e39 None admin] PATH_INFO: `/v3/roles/member` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.199815 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77e52a7-b892-4d42-af06-cf92823b4e39 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-f77e52a7-b892-4d42-af06-cf92823b4e39 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:38.204666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:38.205684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:38.206670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 117/230] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:38.207528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3970d7-3228-4e95-b8ea-e8dd4501f193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.207974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3970d7-3228-4e95-b8ea-e8dd4501f193 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.208283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3970d7-3228-4e95-b8ea-e8dd4501f193 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.208531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77e52a7-b892-4d42-af06-cf92823b4e39 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.208658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3970d7-3228-4e95-b8ea-e8dd4501f193 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.219680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3970d7-3228-4e95-b8ea-e8dd4501f193 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-3a3970d7-3228-4e95-b8ea-e8dd4501f193 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:38.224548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:38.225568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:38.226359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 114/231] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:37:38.227028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3970d7-3228-4e95-b8ea-e8dd4501f193 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.257141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:38.280504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:38.314696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77e52a7-b892-4d42-af06-cf92823b4e39 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.331988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3970d7-3228-4e95-b8ea-e8dd4501f193 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.350795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81526490-11dc-4bea-89a9-f1a2d63ec723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.351420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81526490-11dc-4bea-89a9-f1a2d63ec723 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.351687 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81526490-11dc-4bea-89a9-f1a2d63ec723 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.351956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81526490-11dc-4bea-89a9-f1a2d63ec723 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.353066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81526490-11dc-4bea-89a9-f1a2d63ec723 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:38.354585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81526490-11dc-4bea-89a9-f1a2d63ec723 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:38.356192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 118/232] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:38.356951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81526490-11dc-4bea-89a9-f1a2d63ec723 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.374916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.375543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.375821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.376094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.376817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] RBAC: Adding query filter params (name=member) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:38.377120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:38.378571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:38.388258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 115/233] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:38.389103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.467314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81526490-11dc-4bea-89a9-f1a2d63ec723 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.486690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:38.495407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d871b5-3971-47d2-bc20-8fc74b83997a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.502864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.503443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.503724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.503997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.504691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] RBAC: Adding query filter params (name=reader) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:38.505052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:38.506443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:38.507856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:38.517805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 119/234] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:38.518717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.534433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f1b47e-d412-44fb-b256-6196d49a9d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.535035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1b47e-d412-44fb-b256-6196d49a9d34 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.535310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1b47e-d412-44fb-b256-6196d49a9d34 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.535585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1b47e-d412-44fb-b256-6196d49a9d34 None admin] PATH_INFO: `/v3/groups` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.536237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f1b47e-d412-44fb-b256-6196d49a9d34 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:38.537704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f1b47e-d412-44fb-b256-6196d49a9d34 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:38.553823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 116/235] 10.222.0.39 () {60 vars in 1134 bytes} [Mon Oct 24 08:37:38 2022] POST /identity/v3/groups => generated 229 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:38.557659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f1b47e-d412-44fb-b256-6196d49a9d34 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.635216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb54ac8-e347-450c-832f-984260006199 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.672676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f1b47e-d412-44fb-b256-6196d49a9d34 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.674060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be6222f-7e83-4cd2-a17f-06b9bfa3e1f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.674917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6222f-7e83-4cd2-a17f-06b9bfa3e1f0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.675204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6222f-7e83-4cd2-a17f-06b9bfa3e1f0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.675481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6222f-7e83-4cd2-a17f-06b9bfa3e1f0 None admin] PATH_INFO: `/v3/users/ab6683aa590e4d15b0e747b0707c2ee2` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.676720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be6222f-7e83-4cd2-a17f-06b9bfa3e1f0 None admin] RBAC: Authorizing `identity:get_user(user_id=ab6683aa590e4d15b0e747b0707c2ee2)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:38.678180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be6222f-7e83-4cd2-a17f-06b9bfa3e1f0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:38.679866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 120/236] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:38 2022] GET /identity/v3/users/ab6683aa590e4d15b0e747b0707c2ee2 => generated 293 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:38.686351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be6222f-7e83-4cd2-a17f-06b9bfa3e1f0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.716244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdff19d5-c191-49ea-a0a9-ccff8a6e489c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.716972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff19d5-c191-49ea-a0a9-ccff8a6e489c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.717171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff19d5-c191-49ea-a0a9-ccff8a6e489c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.717443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff19d5-c191-49ea-a0a9-ccff8a6e489c None admin] PATH_INFO: `/v3/users/64d6c21499214689ac9f352f66757e46` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.732279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdff19d5-c191-49ea-a0a9-ccff8a6e489c None admin] RBAC: Authorizing `identity:get_user(user_id=64d6c21499214689ac9f352f66757e46)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:38.733772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdff19d5-c191-49ea-a0a9-ccff8a6e489c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:38.735828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 117/237] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:38 2022] GET /identity/v3/users/64d6c21499214689ac9f352f66757e46 => generated 293 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:38.741252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdff19d5-c191-49ea-a0a9-ccff8a6e489c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:38.759134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:38.763894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:38.808346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be6222f-7e83-4cd2-a17f-06b9bfa3e1f0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.844120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aba7b37-7b36-4570-860d-9e2974719efb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.844915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aba7b37-7b36-4570-860d-9e2974719efb None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.845191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aba7b37-7b36-4570-860d-9e2974719efb None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.845461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aba7b37-7b36-4570-860d-9e2974719efb None admin] PATH_INFO: `/v3/system/users/ab6683aa590e4d15b0e747b0707c2ee2/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.846853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aba7b37-7b36-4570-860d-9e2974719efb None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=ab6683aa590e4d15b0e747b0707c2ee2, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:38.848467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aba7b37-7b36-4570-860d-9e2974719efb None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:38.858436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdff19d5-c191-49ea-a0a9-ccff8a6e489c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:38.867552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 121/238] 10.222.0.39 () {58 vars in 1334 bytes} [Mon Oct 24 08:37:38 2022] PUT /identity/v3/system/users/ab6683aa590e4d15b0e747b0707c2ee2/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:37:38.901267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bad5033-ebc2-4ff0-b4c4-5962e1eb918b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:38.901907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bad5033-ebc2-4ff0-b4c4-5962e1eb918b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:38.902184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bad5033-ebc2-4ff0-b4c4-5962e1eb918b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:38.902450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bad5033-ebc2-4ff0-b4c4-5962e1eb918b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:38.903265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bad5033-ebc2-4ff0-b4c4-5962e1eb918b None admin] RBAC: Adding query filter params (role.id=6d01ecb1b0554e219119728a79f21015, user.id=64d6c21499214689ac9f352f66757e46, scope.system=all) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:38.903571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bad5033-ebc2-4ff0-b4c4-5962e1eb918b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:38.905228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bad5033-ebc2-4ff0-b4c4-5962e1eb918b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:38.911200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 118/239] 10.222.0.39 () {56 vars in 1308 bytes} [Mon Oct 24 08:37:38 2022] GET /identity/v3/role_assignments?user.id=64d6c21499214689ac9f352f66757e46&scope.system=all&role.id=6d01ecb1b0554e219119728a79f21015 => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:40.805922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f535ce-0086-4305-a5a0-25653330287b None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:40.806499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f535ce-0086-4305-a5a0-25653330287b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:40.806575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f535ce-0086-4305-a5a0-25653330287b None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:40.808092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 122/240] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:40.813367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ca0a82-df8b-4487-8c8d-3a866b07d8b4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:40.813829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ca0a82-df8b-4487-8c8d-3a866b07d8b4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:40.813903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ca0a82-df8b-4487-8c8d-3a866b07d8b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:40.832098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-83ca0a82-df8b-4487-8c8d-3a866b07d8b4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:40.833328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-83ca0a82-df8b-4487-8c8d-3a866b07d8b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:40.912598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 119/241] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:41.100197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcae6d-3ef6-4d24-8e77-95638d581e3e None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.100436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcae6d-3ef6-4d24-8e77-95638d581e3e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.100674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcae6d-3ef6-4d24-8e77-95638d581e3e None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.102172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 123/242] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:41.107836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d349dee-4854-4fd9-ae0c-216aca222b81 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.108124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d349dee-4854-4fd9-ae0c-216aca222b81 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.108272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9dcfb7-e502-4f84-8c77-58f7c1b05ce3 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.108443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d349dee-4854-4fd9-ae0c-216aca222b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.108536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9dcfb7-e502-4f84-8c77-58f7c1b05ce3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.108708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9dcfb7-e502-4f84-8c77-58f7c1b05ce3 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.110155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 124/243] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:37:41.115064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63c458-1f8b-40e2-a7df-3982a63d1ed4 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.115352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63c458-1f8b-40e2-a7df-3982a63d1ed4 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.116444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63c458-1f8b-40e2-a7df-3982a63d1ed4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.122380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:41.128724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-0d349dee-4854-4fd9-ae0c-216aca222b81 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:41.129921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0d349dee-4854-4fd9-ae0c-216aca222b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:41.135188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fd63c458-1f8b-40e2-a7df-3982a63d1ed4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:41.136426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fd63c458-1f8b-40e2-a7df-3982a63d1ed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:41.198744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 120/244] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:41.205073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d349dee-4854-4fd9-ae0c-216aca222b81 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:41.209293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 125/245] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:41.307482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d349dee-4854-4fd9-ae0c-216aca222b81 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:41.341416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a9cf9-9803-4544-b5bf-ec172cc07842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:41.342051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a9cf9-9803-4544-b5bf-ec172cc07842 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.342332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a9cf9-9803-4544-b5bf-ec172cc07842 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.342603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a9cf9-9803-4544-b5bf-ec172cc07842 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.343701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674a9cf9-9803-4544-b5bf-ec172cc07842 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:41.345176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674a9cf9-9803-4544-b5bf-ec172cc07842 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:41.346957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 121/246] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:41.352355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd63c458-1f8b-40e2-a7df-3982a63d1ed4 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:41.412277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:41.457209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd63c458-1f8b-40e2-a7df-3982a63d1ed4 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:41.468659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a9cf9-9803-4544-b5bf-ec172cc07842 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:41.472405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:41.494534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73751882-418d-4dd3-918f-f3eefa7fbf1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:41.495145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73751882-418d-4dd3-918f-f3eefa7fbf1b None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.495423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73751882-418d-4dd3-918f-f3eefa7fbf1b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.495696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73751882-418d-4dd3-918f-f3eefa7fbf1b None admin] PATH_INFO: `/v3/groups` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.496371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73751882-418d-4dd3-918f-f3eefa7fbf1b None admin] RBAC: Authorizing `identity:create_group()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:41.497803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73751882-418d-4dd3-918f-f3eefa7fbf1b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:41.515144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 126/247] 10.222.0.39 () {60 vars in 1134 bytes} [Mon Oct 24 08:37:41 2022] POST /identity/v3/groups => generated 226 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:41.515979 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73751882-418d-4dd3-918f-f3eefa7fbf1b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:41.588713 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a9cf9-9803-4544-b5bf-ec172cc07842 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:41.629871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a3c753-9284-451e-8366-ec151432e038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:41.630528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3c753-9284-451e-8366-ec151432e038 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.630799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3c753-9284-451e-8366-ec151432e038 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.631070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3c753-9284-451e-8366-ec151432e038 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.642581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73751882-418d-4dd3-918f-f3eefa7fbf1b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:41.642905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a3c753-9284-451e-8366-ec151432e038 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-62a3c753-9284-451e-8366-ec151432e038 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:41.647913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:41.649045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:41.650056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 122/248] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:37:41.655677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a3c753-9284-451e-8366-ec151432e038 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:41.664901 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:41.692759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260f388b-80aa-4caf-86ac-1ffd1c269a8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:41.693402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f388b-80aa-4caf-86ac-1ffd1c269a8b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.693686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f388b-80aa-4caf-86ac-1ffd1c269a8b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.693952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f388b-80aa-4caf-86ac-1ffd1c269a8b None admin] PATH_INFO: `/v3/roles/member` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.706825 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260f388b-80aa-4caf-86ac-1ffd1c269a8b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-260f388b-80aa-4caf-86ac-1ffd1c269a8b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:41.712513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:41.714231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:41.715189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 127/249] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:41.723662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260f388b-80aa-4caf-86ac-1ffd1c269a8b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:41.754023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:41.797182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a3c753-9284-451e-8366-ec151432e038 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:41.843109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:41.843911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.844331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.844709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.845491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:41.845904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:41.847493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:41.852608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260f388b-80aa-4caf-86ac-1ffd1c269a8b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:41.856840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 123/250] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:41.862550 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:41.897384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:41.897928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:41.898210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:41.898556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:41.899151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] RBAC: Adding query filter params (name=member) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:41.902050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:41.902050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:41.912477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 128/251] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:41.920710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:41.970450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:42.003313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9a7c31-f1ea-4beb-9770-4bb69db3ceb0 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:42.034046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:42.045517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804a998d-6307-4b93-b148-4e5388cbea6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:42.046270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804a998d-6307-4b93-b148-4e5388cbea6b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:42.046552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804a998d-6307-4b93-b148-4e5388cbea6b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:42.046846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804a998d-6307-4b93-b148-4e5388cbea6b None admin] PATH_INFO: `/v3/users/64d6c21499214689ac9f352f66757e46` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:42.047998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804a998d-6307-4b93-b148-4e5388cbea6b None admin] RBAC: Authorizing `identity:get_user(user_id=64d6c21499214689ac9f352f66757e46)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:42.049702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804a998d-6307-4b93-b148-4e5388cbea6b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:42.051590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 124/252] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:41 2022] GET /identity/v3/users/64d6c21499214689ac9f352f66757e46 => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:42.058685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804a998d-6307-4b93-b148-4e5388cbea6b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:42.066119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a29270-0e3b-4a70-99c6-b12b8fc857c5 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:42.104024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafcf033-a4f9-4aa1-9afd-c448473f526f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:42.104767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafcf033-a4f9-4aa1-9afd-c448473f526f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:42.105019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafcf033-a4f9-4aa1-9afd-c448473f526f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:42.105299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafcf033-a4f9-4aa1-9afd-c448473f526f None admin] PATH_INFO: `/v3/users/ab6683aa590e4d15b0e747b0707c2ee2` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:42.106466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafcf033-a4f9-4aa1-9afd-c448473f526f None admin] RBAC: Authorizing `identity:get_user(user_id=ab6683aa590e4d15b0e747b0707c2ee2)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:42.107944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafcf033-a4f9-4aa1-9afd-c448473f526f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:42.110397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 129/253] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:41 2022] GET /identity/v3/users/ab6683aa590e4d15b0e747b0707c2ee2 => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:42.119034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafcf033-a4f9-4aa1-9afd-c448473f526f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:42.194552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804a998d-6307-4b93-b148-4e5388cbea6b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:42.243336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813d2ac5-d53b-4402-8f3e-9bab232a4a31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:42.244461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafcf033-a4f9-4aa1-9afd-c448473f526f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:42.245495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813d2ac5-d53b-4402-8f3e-9bab232a4a31 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:42.246242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813d2ac5-d53b-4402-8f3e-9bab232a4a31 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:42.247670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813d2ac5-d53b-4402-8f3e-9bab232a4a31 None admin] PATH_INFO: `/v3/system/users/64d6c21499214689ac9f352f66757e46/roles/6d01ecb1b0554e219119728a79f21015` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:42.249423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813d2ac5-d53b-4402-8f3e-9bab232a4a31 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=64d6c21499214689ac9f352f66757e46, role_id=6d01ecb1b0554e219119728a79f21015)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:42.251052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813d2ac5-d53b-4402-8f3e-9bab232a4a31 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:42.254514 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:42.268247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 125/254] 10.222.0.39 () {58 vars in 1334 bytes} [Mon Oct 24 08:37:42 2022] PUT /identity/v3/system/users/64d6c21499214689ac9f352f66757e46/roles/6d01ecb1b0554e219119728a79f21015 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:42.288438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5ef47e-cba9-4a54-9603-66eda7cda3a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:42.289066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5ef47e-cba9-4a54-9603-66eda7cda3a2 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:42.289341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5ef47e-cba9-4a54-9603-66eda7cda3a2 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:42.289618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5ef47e-cba9-4a54-9603-66eda7cda3a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:42.290347 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5ef47e-cba9-4a54-9603-66eda7cda3a2 None admin] RBAC: Adding query filter params (role.id=05482458bad348e0be384b75bfd31eac, user.id=ab6683aa590e4d15b0e747b0707c2ee2, scope.system=all) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:42.290659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5ef47e-cba9-4a54-9603-66eda7cda3a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:42.292489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5ef47e-cba9-4a54-9603-66eda7cda3a2 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:42.307640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 130/255] 10.222.0.39 () {56 vars in 1308 bytes} [Mon Oct 24 08:37:42 2022] GET /identity/v3/role_assignments?user.id=ab6683aa590e4d15b0e747b0707c2ee2&scope.system=all&role.id=05482458bad348e0be384b75bfd31eac => generated 510 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:44.151790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16dab0-d357-4b1b-b776-cf25f49de584 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.152334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16dab0-d357-4b1b-b776-cf25f49de584 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.152443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16dab0-d357-4b1b-b776-cf25f49de584 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.153835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 126/256] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:44.159871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79337151-85eb-46b0-bf26-e8b71ca15000 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.160388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79337151-85eb-46b0-bf26-e8b71ca15000 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.160462 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79337151-85eb-46b0-bf26-e8b71ca15000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.189214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-79337151-85eb-46b0-bf26-e8b71ca15000 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:44.190422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-79337151-85eb-46b0-bf26-e8b71ca15000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:44.223331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da346eba-b263-4c9b-827e-c401f12d7159 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.223657 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da346eba-b263-4c9b-827e-c401f12d7159 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.223901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da346eba-b263-4c9b-827e-c401f12d7159 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.225500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 127/257] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:44.236904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85302936-9104-401f-9341-20bfb854a1b0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.237232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85302936-9104-401f-9341-20bfb854a1b0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.237471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85302936-9104-401f-9341-20bfb854a1b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.277271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-85302936-9104-401f-9341-20bfb854a1b0 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:44.278998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-85302936-9104-401f-9341-20bfb854a1b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:44.281624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 131/258] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:44 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 24 08:37:44.290599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 128/259] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:44 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 24 08:37:44.733187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79337151-85eb-46b0-bf26-e8b71ca15000 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:44.746686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79337151-85eb-46b0-bf26-e8b71ca15000 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:44.756847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c19039-f946-4fc6-9b88-9b91cfed4001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:44.757510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c19039-f946-4fc6-9b88-9b91cfed4001 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.757786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c19039-f946-4fc6-9b88-9b91cfed4001 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.758053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c19039-f946-4fc6-9b88-9b91cfed4001 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.770471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c19039-f946-4fc6-9b88-9b91cfed4001 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-71c19039-f946-4fc6-9b88-9b91cfed4001 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:44.777117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:44.778063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:44.778880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:44.780981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 132/260] 10.222.0.39 () {56 vars in 1135 bytes} [Mon Oct 24 08:37:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 46 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:44.785908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85302936-9104-401f-9341-20bfb854a1b0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:44.801388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85302936-9104-401f-9341-20bfb854a1b0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:44.809042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:44.809678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.809955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.810222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.810893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:44.811190 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:44.813024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING py.warnings [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:37:44.813024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: warnings.warn(msg) Oct 24 08:37:44.813024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: Oct 24 08:37:44.813920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:44.829196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 129/261] 10.222.0.39 () {56 vars in 1132 bytes} [Mon Oct 24 08:37:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:44.834776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c19039-f946-4fc6-9b88-9b91cfed4001 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:44.849357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c19039-f946-4fc6-9b88-9b91cfed4001 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:44.854530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:44.855165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.855444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.855713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] PATH_INFO: `/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.866534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] RBAC: Authorizing `identity:get_group(group_id=3a0e0fcfbb6e4e01931f1a784634b201)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:44.868621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:37:44.868621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:37:44.868621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:37:44.870660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:44.872563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 133/262] 10.222.0.39 () {56 vars in 1201 bytes} [Mon Oct 24 08:37:44 2022] GET /identity/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201 => generated 226 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:44.877819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:44.893459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fc994a-14a7-4439-957d-edd03f7a03f2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:44.897941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2f1aa5-a036-45f8-983b-bf53879202e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:44.898576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2f1aa5-a036-45f8-983b-bf53879202e2 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.898853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2f1aa5-a036-45f8-983b-bf53879202e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.899123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2f1aa5-a036-45f8-983b-bf53879202e2 admin admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.900998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2f1aa5-a036-45f8-983b-bf53879202e2 admin admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:44.903687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2f1aa5-a036-45f8-983b-bf53879202e2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:44.905406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 130/263] 10.222.0.39 () {56 vars in 1207 bytes} [Mon Oct 24 08:37:44 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:44.911017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:44.922177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb035c78-2bef-4173-a7ce-ad54d6ccbfdd admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:44.927500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:44.928231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:44.928519 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:44.929013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b/groups/3a0e0fcfbb6e4e01931f1a784634b201/roles/9b956c43ef364a70bf8263ba4314dce7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:44.931513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf4d543d56bb4a978f03e1af565b700b, group_id=3a0e0fcfbb6e4e01931f1a784634b201, role_id=9b956c43ef364a70bf8263ba4314dce7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:44.933009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:37:44.933009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:37:44.933009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:37:44.935275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:44.951065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2f1aa5-a036-45f8-983b-bf53879202e2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:44.965224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2f1aa5-a036-45f8-983b-bf53879202e2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:44.969289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 134/264] 10.222.0.39 () {58 vars in 1463 bytes} [Mon Oct 24 08:37:44 2022] PUT /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b/groups/3a0e0fcfbb6e4e01931f1a784634b201/roles/9b956c43ef364a70bf8263ba4314dce7 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:45.066318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:45.094105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b97d1f1-4ab7-4edf-94bf-06a0c05bf097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:45.094754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97d1f1-4ab7-4edf-94bf-06a0c05bf097 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.095037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97d1f1-4ab7-4edf-94bf-06a0c05bf097 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.095312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97d1f1-4ab7-4edf-94bf-06a0c05bf097 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.116022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97d1f1-4ab7-4edf-94bf-06a0c05bf097 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-1b97d1f1-4ab7-4edf-94bf-06a0c05bf097 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:45.121951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:45.122916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:45.123758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 131/265] 10.222.0.39 () {56 vars in 1120 bytes} [Mon Oct 24 08:37:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:45.129371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:45.145939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73c2236-0c58-486e-875d-4816c4cedb68 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:45.156089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:45.156742 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.157026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.157428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.157987 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] RBAC: Adding query filter params (name=member) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:45.158295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:45.159327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:37:45.159327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:37:45.159327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:37:45.160273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:45.173005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1dc1c9-283f-46b6-9372-65649b076ad7 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.173499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1dc1c9-283f-46b6-9372-65649b076ad7 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.173574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1dc1c9-283f-46b6-9372-65649b076ad7 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.174929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 132/266] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:45.176482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 135/267] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:37:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:45.181142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef0c41-743f-4977-9714-51628b5c9210 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.181940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef0c41-743f-4977-9714-51628b5c9210 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.182214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef0c41-743f-4977-9714-51628b5c9210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.187925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:45.210973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6a417b-0425-4eab-a24d-fbb473c57f92 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:45.216723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-12ef0c41-743f-4977-9714-51628b5c9210 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:45.217934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-12ef0c41-743f-4977-9714-51628b5c9210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:45.225746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0756d16e-6c43-4e61-8b73-4402af392fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:45.226395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756d16e-6c43-4e61-8b73-4402af392fa7 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.226673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756d16e-6c43-4e61-8b73-4402af392fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.226942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756d16e-6c43-4e61-8b73-4402af392fa7 admin admin] PATH_INFO: `/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.232773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0756d16e-6c43-4e61-8b73-4402af392fa7 admin admin] RBAC: Authorizing `identity:get_group(group_id=3a0e0fcfbb6e4e01931f1a784634b201)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:45.234732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0756d16e-6c43-4e61-8b73-4402af392fa7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:45.236472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 136/268] 10.222.0.39 () {56 vars in 1201 bytes} [Mon Oct 24 08:37:45 2022] GET /identity/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201 => generated 226 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:45.242873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0756d16e-6c43-4e61-8b73-4402af392fa7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:45.258778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0756d16e-6c43-4e61-8b73-4402af392fa7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:45.270648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c04914-9284-4ceb-b354-6f8bea6a3af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:45.271299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c04914-9284-4ceb-b354-6f8bea6a3af6 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.271576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c04914-9284-4ceb-b354-6f8bea6a3af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.271849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c04914-9284-4ceb-b354-6f8bea6a3af6 admin admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.273998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c04914-9284-4ceb-b354-6f8bea6a3af6 admin admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:45.276835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c04914-9284-4ceb-b354-6f8bea6a3af6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:45.278664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 137/269] 10.222.0.39 () {56 vars in 1207 bytes} [Mon Oct 24 08:37:45 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:45.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c04914-9284-4ceb-b354-6f8bea6a3af6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:45.321752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c04914-9284-4ceb-b354-6f8bea6a3af6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:45.331001 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f7feee-9fa0-4ac0-8194-11c049fdb3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:45.331664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7feee-9fa0-4ac0-8194-11c049fdb3f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.331945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7feee-9fa0-4ac0-8194-11c049fdb3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.332251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7feee-9fa0-4ac0-8194-11c049fdb3f2 admin admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b/groups/3a0e0fcfbb6e4e01931f1a784634b201/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.335686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7feee-9fa0-4ac0-8194-11c049fdb3f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf4d543d56bb4a978f03e1af565b700b, group_id=3a0e0fcfbb6e4e01931f1a784634b201, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:45.338873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7feee-9fa0-4ac0-8194-11c049fdb3f2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:45.377492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 133/270] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:45.378352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 138/271] 10.222.0.39 () {58 vars in 1463 bytes} [Mon Oct 24 08:37:45 2022] PUT /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b/groups/3a0e0fcfbb6e4e01931f1a784634b201/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:45.564222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07ec975-7ac1-403c-a434-7083b8b0829c None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.564519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07ec975-7ac1-403c-a434-7083b8b0829c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.564717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07ec975-7ac1-403c-a434-7083b8b0829c None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.566357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 134/272] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:45.572577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f382e6d6-3240-41b0-8d7a-02258c1207fd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.572880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f382e6d6-3240-41b0-8d7a-02258c1207fd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.573220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f382e6d6-3240-41b0-8d7a-02258c1207fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.602034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f382e6d6-3240-41b0-8d7a-02258c1207fd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:45.603297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f382e6d6-3240-41b0-8d7a-02258c1207fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:45.664710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:45.679697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 139/273] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:45 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 24 08:37:45.779897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07ec975-7ac1-403c-a434-7083b8b0829c None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:45.913824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07ec975-7ac1-403c-a434-7083b8b0829c None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:45.929746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:45.957891 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade1ce53-db30-405c-858c-246149aea361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:45.958523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade1ce53-db30-405c-858c-246149aea361 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:45.958806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade1ce53-db30-405c-858c-246149aea361 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:45.959082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade1ce53-db30-405c-858c-246149aea361 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:45.971871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade1ce53-db30-405c-858c-246149aea361 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-ade1ce53-db30-405c-858c-246149aea361 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:45.976881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:45.977962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:45.979196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 135/274] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:45 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:37:45.989571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f382e6d6-3240-41b0-8d7a-02258c1207fd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:46.108884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade1ce53-db30-405c-858c-246149aea361 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:46.141166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f382e6d6-3240-41b0-8d7a-02258c1207fd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:46.143342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade1ce53-db30-405c-858c-246149aea361 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:46.163502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06135cc1-ffbf-4e81-ae87-b6b7c8a7d456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:46.164252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06135cc1-ffbf-4e81-ae87-b6b7c8a7d456 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.164469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06135cc1-ffbf-4e81-ae87-b6b7c8a7d456 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.164743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06135cc1-ffbf-4e81-ae87-b6b7c8a7d456 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.187817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:46.189054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.189497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.189915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.190741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] RBAC: Adding query filter params (name=reader) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:46.191195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:46.192817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:46.200828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06135cc1-ffbf-4e81-ae87-b6b7c8a7d456 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:46.203231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 140/275] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-06135cc1-ffbf-4e81-ae87-b6b7c8a7d456 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:46.207181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:46.208353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:46.209390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 136/276] 10.222.0.39 () {56 vars in 1120 bytes} [Mon Oct 24 08:37:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 99 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:46.216192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:46.217433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06135cc1-ffbf-4e81-ae87-b6b7c8a7d456 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:46.231736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06135cc1-ffbf-4e81-ae87-b6b7c8a7d456 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:46.240497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:46.241400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.241889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.242602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.243440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] RBAC: Adding query filter params (name=member) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:46.244476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:46.246556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:46.262591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 137/277] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:37:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:46.269546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:46.276712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:46.293840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c4995e-2c02-407e-93a9-1002e197e16b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:46.300683 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:46.301473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.301750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.302023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] PATH_INFO: `/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.309282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] RBAC: Authorizing `identity:get_group(group_id=3a0e0fcfbb6e4e01931f1a784634b201)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:46.310382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING py.warnings [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:37:46.310382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: warnings.warn(msg) Oct 24 08:37:46.310382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: Oct 24 08:37:46.311852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:46.313576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 138/278] 10.222.0.39 () {56 vars in 1201 bytes} [Mon Oct 24 08:37:46 2022] GET /identity/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201 => generated 226 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:46.319851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:46.321123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc33657-4b25-414d-b07e-176b567a2a8c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:46.334474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f433d53-46f5-4179-b5cd-48c7c25e4efd admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:46.343332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2abf24-a52c-4be5-8c55-a505263ee6f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:46.344103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2abf24-a52c-4be5-8c55-a505263ee6f1 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.344402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2abf24-a52c-4be5-8c55-a505263ee6f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.344729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2abf24-a52c-4be5-8c55-a505263ee6f1 admin admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.346614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2abf24-a52c-4be5-8c55-a505263ee6f1 admin admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:46.348882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2abf24-a52c-4be5-8c55-a505263ee6f1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:46.352363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 139/279] 10.222.0.39 () {56 vars in 1207 bytes} [Mon Oct 24 08:37:46 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:46.357605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2abf24-a52c-4be5-8c55-a505263ee6f1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:46.358800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3048be10-0658-4bf9-8d6b-cdff3278e471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:46.359433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3048be10-0658-4bf9-8d6b-cdff3278e471 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.359710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3048be10-0658-4bf9-8d6b-cdff3278e471 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.359983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3048be10-0658-4bf9-8d6b-cdff3278e471 None admin] PATH_INFO: `/v3/users/64d6c21499214689ac9f352f66757e46` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.361159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3048be10-0658-4bf9-8d6b-cdff3278e471 None admin] RBAC: Authorizing `identity:get_user(user_id=64d6c21499214689ac9f352f66757e46)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:46.362583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3048be10-0658-4bf9-8d6b-cdff3278e471 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:46.364462 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 141/280] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:37:46 2022] GET /identity/v3/users/64d6c21499214689ac9f352f66757e46 => generated 293 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:46.370361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3048be10-0658-4bf9-8d6b-cdff3278e471 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:46.373104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2abf24-a52c-4be5-8c55-a505263ee6f1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:46.378300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae52b84e-caf8-46e7-9b9a-aae9335a08d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:46.378965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae52b84e-caf8-46e7-9b9a-aae9335a08d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.379243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae52b84e-caf8-46e7-9b9a-aae9335a08d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.379564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae52b84e-caf8-46e7-9b9a-aae9335a08d5 admin admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545/groups/3a0e0fcfbb6e4e01931f1a784634b201/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.382775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae52b84e-caf8-46e7-9b9a-aae9335a08d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc9368fb31aa4024a959720dc9541545, group_id=3a0e0fcfbb6e4e01931f1a784634b201, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:46.384260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING py.warnings [None req-ae52b84e-caf8-46e7-9b9a-aae9335a08d5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:37:46.384260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: warnings.warn(msg) Oct 24 08:37:46.384260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: Oct 24 08:37:46.387331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae52b84e-caf8-46e7-9b9a-aae9335a08d5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:46.412332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 140/281] 10.222.0.39 () {58 vars in 1463 bytes} [Mon Oct 24 08:37:46 2022] PUT /identity/v3/projects/dc9368fb31aa4024a959720dc9541545/groups/3a0e0fcfbb6e4e01931f1a784634b201/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:37:46.414813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c7b555-eca3-4eb0-8707-e7d02d890f26 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.415097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c7b555-eca3-4eb0-8707-e7d02d890f26 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.415420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c7b555-eca3-4eb0-8707-e7d02d890f26 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.416945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 141/282] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:46.423954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f742f266-f7db-46f0-a931-9265556b2acc None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.424363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f742f266-f7db-46f0-a931-9265556b2acc None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.424990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f742f266-f7db-46f0-a931-9265556b2acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.461559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f742f266-f7db-46f0-a931-9265556b2acc None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:46.463546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f742f266-f7db-46f0-a931-9265556b2acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:46.485751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3048be10-0658-4bf9-8d6b-cdff3278e471 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:46.519110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:46.528861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:46.529494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:46.529781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:46.530215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:46.530996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] RBAC: Adding query filter params (role.id=6d01ecb1b0554e219119728a79f21015, user.id=64d6c21499214689ac9f352f66757e46, scope.system=all) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:46.531313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:46.534200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:46.552414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 142/283] 10.222.0.39 () {56 vars in 1308 bytes} [Mon Oct 24 08:37:46 2022] GET /identity/v3/role_assignments?user.id=64d6c21499214689ac9f352f66757e46&scope.system=all&role.id=6d01ecb1b0554e219119728a79f21015 => generated 510 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:46.637557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 142/284] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:46 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 24 08:37:47.057465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:47.068285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ec111f-8e86-46a1-a620-c4b8f8000923 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:47.076266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2081ee6e-48ef-44a5-b572-ce916d27a770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:47.076914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2081ee6e-48ef-44a5-b572-ce916d27a770 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:47.077193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2081ee6e-48ef-44a5-b572-ce916d27a770 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:47.077464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2081ee6e-48ef-44a5-b572-ce916d27a770 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:47.090511 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2081ee6e-48ef-44a5-b572-ce916d27a770 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-2081ee6e-48ef-44a5-b572-ce916d27a770 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:47.095389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:47.096312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:47.097109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 143/285] 10.222.0.39 () {56 vars in 1135 bytes} [Mon Oct 24 08:37:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 40 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:47.105008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f742f266-f7db-46f0-a931-9265556b2acc None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:47.117526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f742f266-f7db-46f0-a931-9265556b2acc None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:47.128439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:47.129377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:47.129870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:47.130326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:47.131193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:47.131696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:47.133524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:47.148351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 143/286] 10.222.0.39 () {56 vars in 1132 bytes} [Mon Oct 24 08:37:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:47.155661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2081ee6e-48ef-44a5-b572-ce916d27a770 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:47.166056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2081ee6e-48ef-44a5-b572-ce916d27a770 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:47.176360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079ecf21-1b51-4c5f-b185-e24f27961ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:47.177057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079ecf21-1b51-4c5f-b185-e24f27961ba6 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:47.177338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079ecf21-1b51-4c5f-b185-e24f27961ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:47.177611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079ecf21-1b51-4c5f-b185-e24f27961ba6 admin admin] PATH_INFO: `/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:47.179893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079ecf21-1b51-4c5f-b185-e24f27961ba6 admin admin] RBAC: Authorizing `identity:get_group(group_id=3a0e0fcfbb6e4e01931f1a784634b201)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:47.182352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079ecf21-1b51-4c5f-b185-e24f27961ba6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:47.183958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 144/287] 10.222.0.39 () {56 vars in 1201 bytes} [Mon Oct 24 08:37:47 2022] GET /identity/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201 => generated 226 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:47.190523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:47.205580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21de1b7c-f157-4806-bdde-65c194bbd8a0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:47.216045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f979a5df-519a-43c8-a5b0-c5f4a0affcf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:47.216725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f979a5df-519a-43c8-a5b0-c5f4a0affcf1 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:47.217006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f979a5df-519a-43c8-a5b0-c5f4a0affcf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:47.217282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f979a5df-519a-43c8-a5b0-c5f4a0affcf1 admin admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:47.218443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f979a5df-519a-43c8-a5b0-c5f4a0affcf1 admin admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:47.220567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f979a5df-519a-43c8-a5b0-c5f4a0affcf1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:47.222385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 144/288] 10.222.0.39 () {56 vars in 1207 bytes} [Mon Oct 24 08:37:47 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:47.228168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079ecf21-1b51-4c5f-b185-e24f27961ba6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:47.238959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079ecf21-1b51-4c5f-b185-e24f27961ba6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:47.248315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c516852-52fb-4b33-bbf8-280c5ab90f08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:47.248978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c516852-52fb-4b33-bbf8-280c5ab90f08 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:47.249257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c516852-52fb-4b33-bbf8-280c5ab90f08 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:47.249528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c516852-52fb-4b33-bbf8-280c5ab90f08 admin admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545/groups/3a0e0fcfbb6e4e01931f1a784634b201/roles/9b956c43ef364a70bf8263ba4314dce7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:47.253054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c516852-52fb-4b33-bbf8-280c5ab90f08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc9368fb31aa4024a959720dc9541545, group_id=3a0e0fcfbb6e4e01931f1a784634b201, role_id=9b956c43ef364a70bf8263ba4314dce7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:47.255867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c516852-52fb-4b33-bbf8-280c5ab90f08 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:47.272407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 145/289] 10.222.0.39 () {58 vars in 1463 bytes} [Mon Oct 24 08:37:47 2022] PUT /identity/v3/projects/dc9368fb31aa4024a959720dc9541545/groups/3a0e0fcfbb6e4e01931f1a784634b201/roles/9b956c43ef364a70bf8263ba4314dce7 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:47.737108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9454e9e5-840b-45e3-b50a-a453970905ea None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:47.737434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9454e9e5-840b-45e3-b50a-a453970905ea None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:47.737663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9454e9e5-840b-45e3-b50a-a453970905ea None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:47.739175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 145/290] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:47.746366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a136aa5-06e0-4ac4-99fe-85248eed1402 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:47.746648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a136aa5-06e0-4ac4-99fe-85248eed1402 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:47.746918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a136aa5-06e0-4ac4-99fe-85248eed1402 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:47.776274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-9a136aa5-06e0-4ac4-99fe-85248eed1402 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:47.777494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9a136aa5-06e0-4ac4-99fe-85248eed1402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:47.862532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 146/291] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:47 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 24 08:37:48.427265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9454e9e5-840b-45e3-b50a-a453970905ea None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:48.441561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9454e9e5-840b-45e3-b50a-a453970905ea None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:48.450867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91aea9d-366d-45e0-b58b-c29d7ec6e241 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:48.451377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91aea9d-366d-45e0-b58b-c29d7ec6e241 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:48.451377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91aea9d-366d-45e0-b58b-c29d7ec6e241 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:48.452914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 147/292] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:37:48.454554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8905aadc-f78d-4ed1-b8be-cde81493e1cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:48.455186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8905aadc-f78d-4ed1-b8be-cde81493e1cd admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:48.455466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8905aadc-f78d-4ed1-b8be-cde81493e1cd admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:48.455732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8905aadc-f78d-4ed1-b8be-cde81493e1cd admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:48.457917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5092bb-0c44-4106-8c9b-3c072842d587 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:48.458193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5092bb-0c44-4106-8c9b-3c072842d587 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:48.458461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5092bb-0c44-4106-8c9b-3c072842d587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:48.482648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ae5092bb-0c44-4106-8c9b-3c072842d587 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:48.483838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ae5092bb-0c44-4106-8c9b-3c072842d587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:48.491514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8905aadc-f78d-4ed1-b8be-cde81493e1cd admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-8905aadc-f78d-4ed1-b8be-cde81493e1cd admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:48.502408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:48.503346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:48.504124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 146/293] 10.222.0.39 () {56 vars in 1117 bytes} [Mon Oct 24 08:37:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 77 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:48.522476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8905aadc-f78d-4ed1-b8be-cde81493e1cd admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:48.540376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8905aadc-f78d-4ed1-b8be-cde81493e1cd admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:48.553824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:48.554435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:48.554712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:48.554985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:48.555657 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:48.555958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:48.557817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:48.582438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 147/294] 10.222.0.39 () {56 vars in 1120 bytes} [Mon Oct 24 08:37:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:48.594978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:48.596178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 148/295] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:48.607482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b89719-773b-4974-a099-3b5dd6cf0b34 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:48.615222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85620fca-d857-4e97-8395-d59cb641b1f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:48.615862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85620fca-d857-4e97-8395-d59cb641b1f1 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:48.616157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85620fca-d857-4e97-8395-d59cb641b1f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:48.616442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85620fca-d857-4e97-8395-d59cb641b1f1 admin admin] PATH_INFO: `/v3/groups/c156cf23cfba4dcaa55d09e6ac112242` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:48.624404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85620fca-d857-4e97-8395-d59cb641b1f1 admin admin] RBAC: Authorizing `identity:get_group(group_id=c156cf23cfba4dcaa55d09e6ac112242)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:48.626336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85620fca-d857-4e97-8395-d59cb641b1f1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:48.629271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 148/296] 10.222.0.39 () {56 vars in 1201 bytes} [Mon Oct 24 08:37:48 2022] GET /identity/v3/groups/c156cf23cfba4dcaa55d09e6ac112242 => generated 229 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:48.633955 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5092bb-0c44-4106-8c9b-3c072842d587 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:48.645351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5092bb-0c44-4106-8c9b-3c072842d587 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:48.649981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74af12e0-66c7-4f3e-acdb-61c19c9a5f8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:48.651188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74af12e0-66c7-4f3e-acdb-61c19c9a5f8e admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:48.651754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74af12e0-66c7-4f3e-acdb-61c19c9a5f8e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:48.651754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74af12e0-66c7-4f3e-acdb-61c19c9a5f8e admin admin] PATH_INFO: `/v3/projects/03b1f5372fb94c0fb009656de2a99622` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:48.652855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74af12e0-66c7-4f3e-acdb-61c19c9a5f8e admin admin] RBAC: Authorizing `identity:get_project(project_id=03b1f5372fb94c0fb009656de2a99622)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:48.654653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74af12e0-66c7-4f3e-acdb-61c19c9a5f8e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:48.660272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 149/297] 10.222.0.39 () {56 vars in 1207 bytes} [Mon Oct 24 08:37:48 2022] GET /identity/v3/projects/03b1f5372fb94c0fb009656de2a99622 => generated 344 bytes in 23 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:48.662425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85620fca-d857-4e97-8395-d59cb641b1f1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:48.672788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85620fca-d857-4e97-8395-d59cb641b1f1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:48.677401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2507f4b-fc90-481d-8c0d-0bb26984294d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:48.678060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2507f4b-fc90-481d-8c0d-0bb26984294d admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:48.678344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2507f4b-fc90-481d-8c0d-0bb26984294d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:48.678614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2507f4b-fc90-481d-8c0d-0bb26984294d admin admin] PATH_INFO: `/v3/projects/03b1f5372fb94c0fb009656de2a99622/groups/c156cf23cfba4dcaa55d09e6ac112242/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:48.680694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2507f4b-fc90-481d-8c0d-0bb26984294d admin admin] RBAC: Authorizing `identity:create_grant(project_id=03b1f5372fb94c0fb009656de2a99622, group_id=c156cf23cfba4dcaa55d09e6ac112242, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:48.683773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2507f4b-fc90-481d-8c0d-0bb26984294d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:48.708308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 149/298] 10.222.0.39 () {58 vars in 1463 bytes} [Mon Oct 24 08:37:48 2022] PUT /identity/v3/projects/03b1f5372fb94c0fb009656de2a99622/groups/c156cf23cfba4dcaa55d09e6ac112242/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:37:49.061785 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a16fa10-01ee-478f-bccd-88cf8758834a None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:49.062117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a16fa10-01ee-478f-bccd-88cf8758834a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:49.062336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a16fa10-01ee-478f-bccd-88cf8758834a None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:49.064049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 150/299] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:49.067010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2507f4b-fc90-481d-8c0d-0bb26984294d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:49.086974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:49.092813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1205824-4165-4989-820f-3c9e8b1eceb9 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:49.093111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1205824-4165-4989-820f-3c9e8b1eceb9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:49.093365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1205824-4165-4989-820f-3c9e8b1eceb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:49.125209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b1205824-4165-4989-820f-3c9e8b1eceb9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:49.126431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b1205824-4165-4989-820f-3c9e8b1eceb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:49.187332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:49.206655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2507f4b-fc90-481d-8c0d-0bb26984294d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:49.255336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9889a690-4bfb-406e-a406-c18c18491d7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:49.256192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9889a690-4bfb-406e-a406-c18c18491d7d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:49.256475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9889a690-4bfb-406e-a406-c18c18491d7d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:49.256749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9889a690-4bfb-406e-a406-c18c18491d7d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:49.276510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 151/300] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:49.277946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa53ec1-92a4-47bf-a47e-f3f8205e81fc None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:49.278229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa53ec1-92a4-47bf-a47e-f3f8205e81fc None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:49.278499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa53ec1-92a4-47bf-a47e-f3f8205e81fc None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:49.279071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9889a690-4bfb-406e-a406-c18c18491d7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:49.279788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 152/301] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-9889a690-4bfb-406e-a406-c18c18491d7d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:49.285768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:49.286884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:49.287897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 150/302] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:37:49 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:49.293605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c58623-1432-4df3-ac00-4db5c0012c1d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:49.294083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c58623-1432-4df3-ac00-4db5c0012c1d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:49.294496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c58623-1432-4df3-ac00-4db5c0012c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:49.300428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9889a690-4bfb-406e-a406-c18c18491d7d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:49.317083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-24c58623-1432-4df3-ac00-4db5c0012c1d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:49.318529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-24c58623-1432-4df3-ac00-4db5c0012c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:49.420511 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 153/303] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:49.454378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:49.483206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9889a690-4bfb-406e-a406-c18c18491d7d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:49.525102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:49.525715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:49.525992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:49.526264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:49.526932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:49.527235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:49.528673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:49.545026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 151/304] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:37:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:49.550771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c58623-1432-4df3-ac00-4db5c0012c1d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:49.598600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:49.661937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:49.691772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c58623-1432-4df3-ac00-4db5c0012c1d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:49.788589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cfee0c-46ba-4615-8a28-5f9e1a719d4f None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:49.796451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bc2ec8-1f91-4f2b-9eb7-48caf952d7cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:49.797272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bc2ec8-1f91-4f2b-9eb7-48caf952d7cd None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:49.798404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bc2ec8-1f91-4f2b-9eb7-48caf952d7cd None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:49.798404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bc2ec8-1f91-4f2b-9eb7-48caf952d7cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:49.842353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f5a531-31f4-497a-8341-4cd1cf70a804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:49.843002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5a531-31f4-497a-8341-4cd1cf70a804 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:49.843279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5a531-31f4-497a-8341-4cd1cf70a804 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:49.843548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5a531-31f4-497a-8341-4cd1cf70a804 None admin] PATH_INFO: `/v3/roles/member` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:49.851747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:49.859388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f5a531-31f4-497a-8341-4cd1cf70a804 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-96f5a531-31f4-497a-8341-4cd1cf70a804 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:49.864973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:49.866352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:49.867183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 152/305] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:49.872179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f5a531-31f4-497a-8341-4cd1cf70a804 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:49.959380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bc2ec8-1f91-4f2b-9eb7-48caf952d7cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:49.962036 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bc2ec8-1f91-4f2b-9eb7-48caf952d7cd None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:50.045835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f5a531-31f4-497a-8341-4cd1cf70a804 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:50.105625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e7e01f-f883-43fe-9f56-940224eff394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:50.106653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e7e01f-f883-43fe-9f56-940224eff394 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:50.107132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e7e01f-f883-43fe-9f56-940224eff394 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:50.107599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e7e01f-f883-43fe-9f56-940224eff394 None admin] PATH_INFO: `/v3/roles/member` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:50.111377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:50.123746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e7e01f-f883-43fe-9f56-940224eff394 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-d2e7e01f-f883-43fe-9f56-940224eff394 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:50.129705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:50.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 24 08:37:50.131413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:50.132719 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 154/306] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 581 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:50.132719 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bc2ec8-1f91-4f2b-9eb7-48caf952d7cd None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:50.134414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 153/307] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:50.140596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e7e01f-f883-43fe-9f56-940224eff394 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:50.210574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:50.248735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bc2ec8-1f91-4f2b-9eb7-48caf952d7cd None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:50.262235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e7e01f-f883-43fe-9f56-940224eff394 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:50.290292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:50.290956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:50.291310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:50.291583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:50.292447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] RBAC: Adding query filter params (name=member) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:50.292756 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:50.294163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:50.303305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 155/308] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:50.304094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:50.310088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0ab02c-e5d7-445d-9b5d-c8d55a8010db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:50.311043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0ab02c-e5d7-445d-9b5d-c8d55a8010db None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:50.311607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0ab02c-e5d7-445d-9b5d-c8d55a8010db None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:50.312084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0ab02c-e5d7-445d-9b5d-c8d55a8010db None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:50.313801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0ab02c-e5d7-445d-9b5d-c8d55a8010db None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:50.315486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0ab02c-e5d7-445d-9b5d-c8d55a8010db None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:50.317759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 154/309] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:50 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:50.318792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0ab02c-e5d7-445d-9b5d-c8d55a8010db None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:50.388528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:50.438014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0891be66-9dce-4ae6-b982-6200516d6c26 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:50.448919 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0ab02c-e5d7-445d-9b5d-c8d55a8010db None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:50.490521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:50.501474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:50.502094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:50.502374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:50.502648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:50.503301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] RBAC: Adding query filter params (name=member) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:50.503601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:50.504480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e49e7e3-9354-4b89-8fcf-c235f9d3719a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:50.505102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:50.505275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e49e7e3-9354-4b89-8fcf-c235f9d3719a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:50.505582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e49e7e3-9354-4b89-8fcf-c235f9d3719a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:50.505894 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e49e7e3-9354-4b89-8fcf-c235f9d3719a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:50.519371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 156/310] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:37:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:50.520364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:50.595571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e49e7e3-9354-4b89-8fcf-c235f9d3719a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:50.597229 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e49e7e3-9354-4b89-8fcf-c235f9d3719a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:50.684342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f96496-3607-4af9-988c-1f66827a6784 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:50.708303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:50.729907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 155/311] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:50.730787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e49e7e3-9354-4b89-8fcf-c235f9d3719a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:50.739581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1aa8250-2fbe-480c-b2d1-4b2e135e55ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:50.740323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aa8250-2fbe-480c-b2d1-4b2e135e55ef None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:50.740602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aa8250-2fbe-480c-b2d1-4b2e135e55ef None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:50.740873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aa8250-2fbe-480c-b2d1-4b2e135e55ef None admin] PATH_INFO: `/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:50.742006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1aa8250-2fbe-480c-b2d1-4b2e135e55ef None admin] RBAC: Authorizing `identity:get_group(group_id=3a0e0fcfbb6e4e01931f1a784634b201)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:50.743722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1aa8250-2fbe-480c-b2d1-4b2e135e55ef None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:50.746635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 157/312] 10.222.0.39 () {56 vars in 1180 bytes} [Mon Oct 24 08:37:50 2022] GET /identity/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201 => generated 226 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:50.747536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1aa8250-2fbe-480c-b2d1-4b2e135e55ef None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:50.771614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:50.873918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e49e7e3-9354-4b89-8fcf-c235f9d3719a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:50.895764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1aa8250-2fbe-480c-b2d1-4b2e135e55ef None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:50.922406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5252ffa-a6c6-4b41-9d97-cce35983f9b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:50.923183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5252ffa-a6c6-4b41-9d97-cce35983f9b3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:50.923564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5252ffa-a6c6-4b41-9d97-cce35983f9b3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:50.923949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5252ffa-a6c6-4b41-9d97-cce35983f9b3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:50.950818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da5b2e6-c76c-427d-a53b-a138a422fe22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:50.951894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da5b2e6-c76c-427d-a53b-a138a422fe22 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:50.951894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da5b2e6-c76c-427d-a53b-a138a422fe22 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:50.951894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da5b2e6-c76c-427d-a53b-a138a422fe22 None admin] PATH_INFO: `/v3/projects/cf4d543d56bb4a978f03e1af565b700b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:50.955566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da5b2e6-c76c-427d-a53b-a138a422fe22 None admin] RBAC: Authorizing `identity:get_project(project_id=cf4d543d56bb4a978f03e1af565b700b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:50.955566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da5b2e6-c76c-427d-a53b-a138a422fe22 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:50.956781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 158/313] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:50 2022] GET /identity/v3/projects/cf4d543d56bb4a978f03e1af565b700b => generated 298 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:50.957714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da5b2e6-c76c-427d-a53b-a138a422fe22 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:51.038691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5252ffa-a6c6-4b41-9d97-cce35983f9b3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:51.040517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5252ffa-a6c6-4b41-9d97-cce35983f9b3 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:51.048279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:51.075141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:51.136207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da5b2e6-c76c-427d-a53b-a138a422fe22 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:51.180161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:51.180760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.181038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.181309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.185354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] RBAC: Adding query filter params (group.id=3a0e0fcfbb6e4e01931f1a784634b201, role.id=9b956c43ef364a70bf8263ba4314dce7, scope.project.id=cf4d543d56bb4a978f03e1af565b700b) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:51.185354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:51.186724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:51.190571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 156/314] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:51.192851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7166fad-5d20-47e3-bad0-98e17853b474 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.193131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7166fad-5d20-47e3-bad0-98e17853b474 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.193404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7166fad-5d20-47e3-bad0-98e17853b474 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.194876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 157/315] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:51.195541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7166fad-5d20-47e3-bad0-98e17853b474 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:51.196006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 159/316] 10.222.0.39 () {56 vars in 1376 bytes} [Mon Oct 24 08:37:50 2022] GET /identity/v3/role_assignments?group.id=3a0e0fcfbb6e4e01931f1a784634b201&scope.project.id=cf4d543d56bb4a978f03e1af565b700b&role.id=9b956c43ef364a70bf8263ba4314dce7 => generated 611 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:51.201564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:51.345840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7166fad-5d20-47e3-bad0-98e17853b474 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:51.365292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c8826f-3589-4a41-a021-aa2747a54a57 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:51.386608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:51.393918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:51.401557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967015bf-3037-4ec9-a28b-d55ad9a9e30b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:51.402634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967015bf-3037-4ec9-a28b-d55ad9a9e30b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.403288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967015bf-3037-4ec9-a28b-d55ad9a9e30b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.404124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967015bf-3037-4ec9-a28b-d55ad9a9e30b None admin] PATH_INFO: `/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.406743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967015bf-3037-4ec9-a28b-d55ad9a9e30b None admin] RBAC: Authorizing `identity:get_group(group_id=3a0e0fcfbb6e4e01931f1a784634b201)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:51.408923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967015bf-3037-4ec9-a28b-d55ad9a9e30b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:51.411774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 158/317] 10.222.0.39 () {56 vars in 1180 bytes} [Mon Oct 24 08:37:51 2022] GET /identity/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201 => generated 226 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:51.414368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41895615-cd3d-42f4-9bac-b51847e4ae95 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.414652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41895615-cd3d-42f4-9bac-b51847e4ae95 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.414923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41895615-cd3d-42f4-9bac-b51847e4ae95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.425369 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b436a-ebdb-4374-accd-964ed1761d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:51.426395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b436a-ebdb-4374-accd-964ed1761d96 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.426909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b436a-ebdb-4374-accd-964ed1761d96 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.427430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b436a-ebdb-4374-accd-964ed1761d96 None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.429223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b436a-ebdb-4374-accd-964ed1761d96 None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:51.431040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b436a-ebdb-4374-accd-964ed1761d96 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:51.435882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-41895615-cd3d-42f4-9bac-b51847e4ae95 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:51.437221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-41895615-cd3d-42f4-9bac-b51847e4ae95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:51.437728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 160/318] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:51 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:51.438823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b436a-ebdb-4374-accd-964ed1761d96 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:51.538908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 159/319] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:51.539931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41895615-cd3d-42f4-9bac-b51847e4ae95 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:51.581208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b436a-ebdb-4374-accd-964ed1761d96 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:51.623618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c9dad2-db55-46ea-95b6-06a2df2b8ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:51.624277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c9dad2-db55-46ea-95b6-06a2df2b8ab4 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.624996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c9dad2-db55-46ea-95b6-06a2df2b8ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.624996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c9dad2-db55-46ea-95b6-06a2df2b8ab4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.625541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c9dad2-db55-46ea-95b6-06a2df2b8ab4 None admin] RBAC: Adding query filter params (group.id=3a0e0fcfbb6e4e01931f1a784634b201, role.id=05482458bad348e0be384b75bfd31eac, scope.project.id=cf4d543d56bb4a978f03e1af565b700b) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:51.625850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c9dad2-db55-46ea-95b6-06a2df2b8ab4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:51.628163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c9dad2-db55-46ea-95b6-06a2df2b8ab4 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:51.636245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 161/320] 10.222.0.39 () {56 vars in 1376 bytes} [Mon Oct 24 08:37:51 2022] GET /identity/v3/role_assignments?group.id=3a0e0fcfbb6e4e01931f1a784634b201&scope.project.id=cf4d543d56bb4a978f03e1af565b700b&role.id=05482458bad348e0be384b75bfd31eac => generated 611 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:51.637703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa82e4d6-1056-4e79-9f02-23ba0d08f847 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.637932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa82e4d6-1056-4e79-9f02-23ba0d08f847 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.638208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa82e4d6-1056-4e79-9f02-23ba0d08f847 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.643912 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 162/321] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:51.645880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539e4e0a-f283-412f-acc2-d30975c6667c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.646179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539e4e0a-f283-412f-acc2-d30975c6667c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.646436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539e4e0a-f283-412f-acc2-d30975c6667c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.669632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-539e4e0a-f283-412f-acc2-d30975c6667c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:51.670901 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-539e4e0a-f283-412f-acc2-d30975c6667c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:51.679258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41895615-cd3d-42f4-9bac-b51847e4ae95 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:51.710581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:51.727147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf81e01-1c73-4214-9bed-f19f8cd7c89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:51.728226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81e01-1c73-4214-9bed-f19f8cd7c89b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.729284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81e01-1c73-4214-9bed-f19f8cd7c89b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.729781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81e01-1c73-4214-9bed-f19f8cd7c89b None admin] PATH_INFO: `/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.731537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf81e01-1c73-4214-9bed-f19f8cd7c89b None admin] RBAC: Authorizing `identity:get_group(group_id=3a0e0fcfbb6e4e01931f1a784634b201)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:51.733273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf81e01-1c73-4214-9bed-f19f8cd7c89b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:51.735024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 160/322] 10.222.0.39 () {56 vars in 1180 bytes} [Mon Oct 24 08:37:51 2022] GET /identity/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:51.739252 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:51.745060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf81e01-1c73-4214-9bed-f19f8cd7c89b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:51.788479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 163/323] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:51.859892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539e4e0a-f283-412f-acc2-d30975c6667c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:51.910746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf81e01-1c73-4214-9bed-f19f8cd7c89b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:51.969598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:51.970416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:51.970717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:51.971013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:51.971835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] RBAC: Adding query filter params (group.id=3a0e0fcfbb6e4e01931f1a784634b201, role.id=05482458bad348e0be384b75bfd31eac, scope.project.id=dc9368fb31aa4024a959720dc9541545) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:51.972299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:51.973641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:51.981093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 161/324] 10.222.0.39 () {56 vars in 1376 bytes} [Mon Oct 24 08:37:51 2022] GET /identity/v3/role_assignments?group.id=3a0e0fcfbb6e4e01931f1a784634b201&scope.project.id=dc9368fb31aa4024a959720dc9541545&role.id=05482458bad348e0be384b75bfd31eac => generated 611 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:52.044208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539e4e0a-f283-412f-acc2-d30975c6667c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:52.091634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875b8ba-3756-4f3a-b744-98623cb5593c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:52.092377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875b8ba-3756-4f3a-b744-98623cb5593c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:52.092660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875b8ba-3756-4f3a-b744-98623cb5593c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:52.092938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875b8ba-3756-4f3a-b744-98623cb5593c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:52.108416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a875b8ba-3756-4f3a-b744-98623cb5593c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-a875b8ba-3756-4f3a-b744-98623cb5593c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:52.113858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:52.116012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:52.117951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:52.118877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 164/325] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:37:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:37:52.126388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875b8ba-3756-4f3a-b744-98623cb5593c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:52.183466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:52.207027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:52.273500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97960051-1aaf-436b-a40c-a8170a783e1f None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:52.284077 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875b8ba-3756-4f3a-b744-98623cb5593c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:52.318792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5f25d9-eaf1-47c0-a85e-4dbce90c573e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:52.319421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f25d9-eaf1-47c0-a85e-4dbce90c573e None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:52.319830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f25d9-eaf1-47c0-a85e-4dbce90c573e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:52.319985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f25d9-eaf1-47c0-a85e-4dbce90c573e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:52.334548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:52.335170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:52.335454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:52.337316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:52.337316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:52.337316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:52.339202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:52.339935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5f25d9-eaf1-47c0-a85e-4dbce90c573e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-ea5f25d9-eaf1-47c0-a85e-4dbce90c573e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:52.346674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:52.348250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:37:52.349232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:52.350002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 165/327] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:37:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:52.350086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 162/327] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:37:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:52.354624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5f25d9-eaf1-47c0-a85e-4dbce90c573e None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:52.355954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:52.482145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:52.493334 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2583ec3-9880-4c04-97b9-54d548a53198 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:52.498419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5f25d9-eaf1-47c0-a85e-4dbce90c573e None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:52.515911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:52.544271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:52.544499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e52e7-b74f-4e8b-8c0e-4cd2050caa9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:52.544917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e52e7-b74f-4e8b-8c0e-4cd2050caa9d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:52.544991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:52.545227 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e52e7-b74f-4e8b-8c0e-4cd2050caa9d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:52.545322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:52.545482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:52.545584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e52e7-b74f-4e8b-8c0e-4cd2050caa9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:52.546112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:52.546421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:52.547807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:52.559894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 166/328] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:37:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:52.565586 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:52.639941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1e52e7-b74f-4e8b-8c0e-4cd2050caa9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:52.641630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1e52e7-b74f-4e8b-8c0e-4cd2050caa9d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:52.704709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cebd128-3ec9-4134-bc9b-45c2313b9ac0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:52.744144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10790710-2ddc-4d67-883d-e9cb5ba2b588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:52.744859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10790710-2ddc-4d67-883d-e9cb5ba2b588 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:52.744971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10790710-2ddc-4d67-883d-e9cb5ba2b588 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:52.745228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10790710-2ddc-4d67-883d-e9cb5ba2b588 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:52.769841 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 163/329] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:37:52.774816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:52.776421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e52e7-b74f-4e8b-8c0e-4cd2050caa9d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:52.794653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:52.819900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10790710-2ddc-4d67-883d-e9cb5ba2b588 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:52.821402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10790710-2ddc-4d67-883d-e9cb5ba2b588 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:52.882343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e52e7-b74f-4e8b-8c0e-4cd2050caa9d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:52.923480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965530fc-bcbf-445d-89b8-3967190375a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:52.924074 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965530fc-bcbf-445d-89b8-3967190375a1 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:52.924359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965530fc-bcbf-445d-89b8-3967190375a1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:52.924620 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965530fc-bcbf-445d-89b8-3967190375a1 None admin] PATH_INFO: `/v3/projects/dc9368fb31aa4024a959720dc9541545` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:52.925740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965530fc-bcbf-445d-89b8-3967190375a1 None admin] RBAC: Authorizing `identity:get_project(project_id=dc9368fb31aa4024a959720dc9541545)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:52.927242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965530fc-bcbf-445d-89b8-3967190375a1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:52.929174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 164/330] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:52 2022] GET /identity/v3/projects/dc9368fb31aa4024a959720dc9541545 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:52.932328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 167/331] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:37:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 24 08:37:52.934873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965530fc-bcbf-445d-89b8-3967190375a1 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:52.938678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10790710-2ddc-4d67-883d-e9cb5ba2b588 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:53.010962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:53.035862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:53.057310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10790710-2ddc-4d67-883d-e9cb5ba2b588 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:53.057549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965530fc-bcbf-445d-89b8-3967190375a1 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:53.095786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98827cad-826c-4139-9cc5-0079038ff88c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:53.096732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98827cad-826c-4139-9cc5-0079038ff88c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:53.097020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98827cad-826c-4139-9cc5-0079038ff88c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:53.097292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98827cad-826c-4139-9cc5-0079038ff88c None admin] PATH_INFO: `/v3/projects/03b1f5372fb94c0fb009656de2a99622` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:53.098428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98827cad-826c-4139-9cc5-0079038ff88c None admin] RBAC: Authorizing `identity:get_project(project_id=03b1f5372fb94c0fb009656de2a99622)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:53.099871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98827cad-826c-4139-9cc5-0079038ff88c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:53.101325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e794fc-102e-4e61-a97d-96fc74dc0a2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:53.101836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 168/332] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:37:52 2022] GET /identity/v3/projects/03b1f5372fb94c0fb009656de2a99622 => generated 344 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:53.101959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e794fc-102e-4e61-a97d-96fc74dc0a2a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:53.102218 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e794fc-102e-4e61-a97d-96fc74dc0a2a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:53.102496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e794fc-102e-4e61-a97d-96fc74dc0a2a None admin] PATH_INFO: `/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:53.103737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e794fc-102e-4e61-a97d-96fc74dc0a2a None admin] RBAC: Authorizing `identity:get_group(group_id=3a0e0fcfbb6e4e01931f1a784634b201)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:53.105331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e794fc-102e-4e61-a97d-96fc74dc0a2a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:53.111783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 165/333] 10.222.0.39 () {56 vars in 1180 bytes} [Mon Oct 24 08:37:52 2022] GET /identity/v3/groups/3a0e0fcfbb6e4e01931f1a784634b201 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:53.111964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98827cad-826c-4139-9cc5-0079038ff88c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:53.114601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e794fc-102e-4e61-a97d-96fc74dc0a2a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:53.221912 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98827cad-826c-4139-9cc5-0079038ff88c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:53.228150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e794fc-102e-4e61-a97d-96fc74dc0a2a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:53.248629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:53.257746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eedf37-c994-4577-bf12-6a4b73791a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:53.258368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eedf37-c994-4577-bf12-6a4b73791a06 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:53.258654 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eedf37-c994-4577-bf12-6a4b73791a06 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:53.258926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eedf37-c994-4577-bf12-6a4b73791a06 None admin] PATH_INFO: `/v3/groups/c156cf23cfba4dcaa55d09e6ac112242` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:53.260836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70eedf37-c994-4577-bf12-6a4b73791a06 None admin] RBAC: Authorizing `identity:get_group(group_id=c156cf23cfba4dcaa55d09e6ac112242)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:53.262326 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70eedf37-c994-4577-bf12-6a4b73791a06 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:53.264186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 169/334] 10.222.0.39 () {56 vars in 1180 bytes} [Mon Oct 24 08:37:53 2022] GET /identity/v3/groups/c156cf23cfba4dcaa55d09e6ac112242 => generated 229 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:53.264815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9e0596-a1ab-4897-90f0-574862b74df6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:53.265495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9e0596-a1ab-4897-90f0-574862b74df6 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:53.265778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9e0596-a1ab-4897-90f0-574862b74df6 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:53.266047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9e0596-a1ab-4897-90f0-574862b74df6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:53.266770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9e0596-a1ab-4897-90f0-574862b74df6 None admin] RBAC: Adding query filter params (group.id=3a0e0fcfbb6e4e01931f1a784634b201, role.id=9b956c43ef364a70bf8263ba4314dce7, scope.project.id=dc9368fb31aa4024a959720dc9541545) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:53.267077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9e0596-a1ab-4897-90f0-574862b74df6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:53.268523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9e0596-a1ab-4897-90f0-574862b74df6 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:53.269971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eedf37-c994-4577-bf12-6a4b73791a06 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:53.274915 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 166/335] 10.222.0.39 () {56 vars in 1376 bytes} [Mon Oct 24 08:37:53 2022] GET /identity/v3/role_assignments?group.id=3a0e0fcfbb6e4e01931f1a784634b201&scope.project.id=dc9368fb31aa4024a959720dc9541545&role.id=9b956c43ef364a70bf8263ba4314dce7 => generated 611 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:53.376008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eedf37-c994-4577-bf12-6a4b73791a06 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:53.411644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528e7e84-e60f-4a0c-b457-6a17bb578734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:53.412292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e7e84-e60f-4a0c-b457-6a17bb578734 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:53.412576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e7e84-e60f-4a0c-b457-6a17bb578734 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:53.412855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e7e84-e60f-4a0c-b457-6a17bb578734 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:53.413589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528e7e84-e60f-4a0c-b457-6a17bb578734 None admin] RBAC: Adding query filter params (group.id=c156cf23cfba4dcaa55d09e6ac112242, role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=03b1f5372fb94c0fb009656de2a99622) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:53.413897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528e7e84-e60f-4a0c-b457-6a17bb578734 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:53.415321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528e7e84-e60f-4a0c-b457-6a17bb578734 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:53.422493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 170/336] 10.222.0.39 () {56 vars in 1376 bytes} [Mon Oct 24 08:37:53 2022] GET /identity/v3/role_assignments?group.id=c156cf23cfba4dcaa55d09e6ac112242&scope.project.id=03b1f5372fb94c0fb009656de2a99622&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 611 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:56.223298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f211e1f-994b-4c18-a229-d1d0b6a79e41 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:56.223851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f211e1f-994b-4c18-a229-d1d0b6a79e41 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:56.223851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f211e1f-994b-4c18-a229-d1d0b6a79e41 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:56.225597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 167/337] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:56.232568 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89674ee2-c4d6-4980-8754-270658c0d7d1 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:56.233124 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89674ee2-c4d6-4980-8754-270658c0d7d1 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:56.233124 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89674ee2-c4d6-4980-8754-270658c0d7d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:56.265512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-89674ee2-c4d6-4980-8754-270658c0d7d1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:56.266655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-89674ee2-c4d6-4980-8754-270658c0d7d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:56.346924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:56.363857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 171/338] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:37:56.389432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe54323-9694-451c-b2ba-985d9463a26a None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:56.389732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe54323-9694-451c-b2ba-985d9463a26a None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:56.389990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe54323-9694-451c-b2ba-985d9463a26a None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:56.392860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 168/339] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:56.398189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe3690e-7fa4-4766-9a4b-7c3263c3d513 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:56.398497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe3690e-7fa4-4766-9a4b-7c3263c3d513 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:56.398809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe3690e-7fa4-4766-9a4b-7c3263c3d513 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:56.427399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-2fe3690e-7fa4-4766-9a4b-7c3263c3d513 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:56.428613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2fe3690e-7fa4-4766-9a4b-7c3263c3d513 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:56.515822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 172/340] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:56.750893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe54323-9694-451c-b2ba-985d9463a26a None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:56.771765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:56.817156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a4b2fb-434a-4b7f-93cf-5b21acb1229f None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:56.817488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a4b2fb-434a-4b7f-93cf-5b21acb1229f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:56.817768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a4b2fb-434a-4b7f-93cf-5b21acb1229f None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:56.819405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 173/341] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:56.824092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a4b2fb-434a-4b7f-93cf-5b21acb1229f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:56.895041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe54323-9694-451c-b2ba-985d9463a26a None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:56.953953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed28a20-2dd6-4d71-a231-a206a5c53f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:56.954958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed28a20-2dd6-4d71-a231-a206a5c53f00 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:56.955506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed28a20-2dd6-4d71-a231-a206a5c53f00 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:56.955964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed28a20-2dd6-4d71-a231-a206a5c53f00 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:56.968219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed28a20-2dd6-4d71-a231-a206a5c53f00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:56.968382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a4b2fb-434a-4b7f-93cf-5b21acb1229f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:56.971185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed28a20-2dd6-4d71-a231-a206a5c53f00 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:56.987321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-3ed28a20-2dd6-4d71-a231-a206a5c53f00 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:37:57.002016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:57.002972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:37:57.003827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:37:57.003827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:37:57.003827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:57.004167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 169/342] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:57.005924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f1ae5-4c6c-4cfe-b772-1154f4eb58f3 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.006047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:57.006376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f1ae5-4c6c-4cfe-b772-1154f4eb58f3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.006604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.006830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f1ae5-4c6c-4cfe-b772-1154f4eb58f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.006946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.009767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.018472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:57.020009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:57.032183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7b0f1ae5-4c6c-4cfe-b772-1154f4eb58f3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:57.033816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7b0f1ae5-4c6c-4cfe-b772-1154f4eb58f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:37:57.037233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:57.039716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:37:57.042426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:37:57.042426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:37:57.042426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:57.042426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 174/343] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:57.042426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:57.113403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:57.124773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 170/344] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:57.124773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f1ae5-4c6c-4cfe-b772-1154f4eb58f3 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:57.185175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8e5b3f-c624-4c7b-9374-837e8efa39ad None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:57.238004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb14375e-87de-488a-90f7-a4587ee94b31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:57.239460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb14375e-87de-488a-90f7-a4587ee94b31 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.239906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb14375e-87de-488a-90f7-a4587ee94b31 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.240360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb14375e-87de-488a-90f7-a4587ee94b31 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.241170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb14375e-87de-488a-90f7-a4587ee94b31 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:57.241717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb14375e-87de-488a-90f7-a4587ee94b31 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:57.243675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb14375e-87de-488a-90f7-a4587ee94b31 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:57.246432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f1ae5-4c6c-4cfe-b772-1154f4eb58f3 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:57.268874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 175/345] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:37:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:57.270673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0116183f-f76e-40b1-a3a7-496601b09056 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.271184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0116183f-f76e-40b1-a3a7-496601b09056 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.271692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0116183f-f76e-40b1-a3a7-496601b09056 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.273357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 176/346] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:57.274591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0116183f-f76e-40b1-a3a7-496601b09056 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:57.288679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784a6492-a99f-4986-90e7-b9bf34461690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:57.288679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a6492-a99f-4986-90e7-b9bf34461690 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.288679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a6492-a99f-4986-90e7-b9bf34461690 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.289023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a6492-a99f-4986-90e7-b9bf34461690 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.289632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784a6492-a99f-4986-90e7-b9bf34461690 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:57.289943 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784a6492-a99f-4986-90e7-b9bf34461690 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:57.294850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784a6492-a99f-4986-90e7-b9bf34461690 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:57.306129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 171/347] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:37:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:57.307479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f9044-2f8a-4c66-b28f-02af8dcf2a8a None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.307746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f9044-2f8a-4c66-b28f-02af8dcf2a8a None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.308105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f9044-2f8a-4c66-b28f-02af8dcf2a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.325280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:57.329061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b00f9044-2f8a-4c66-b28f-02af8dcf2a8a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:57.330853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b00f9044-2f8a-4c66-b28f-02af8dcf2a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:57.392350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:57.408475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 172/348] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:57.409433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f9044-2f8a-4c66-b28f-02af8dcf2a8a None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:57.426231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0116183f-f76e-40b1-a3a7-496601b09056 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:57.475863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be897017-cf99-4fdb-a482-14fd6a0e707b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:57.476977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be897017-cf99-4fdb-a482-14fd6a0e707b None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.477557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be897017-cf99-4fdb-a482-14fd6a0e707b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.478027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be897017-cf99-4fdb-a482-14fd6a0e707b None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.479011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be897017-cf99-4fdb-a482-14fd6a0e707b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:57.480811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be897017-cf99-4fdb-a482-14fd6a0e707b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:57.515297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 177/349] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:57 2022] POST /identity/v3/users => generated 246 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:37:57.516521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be897017-cf99-4fdb-a482-14fd6a0e707b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:57.543097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f9044-2f8a-4c66-b28f-02af8dcf2a8a None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:57.642623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3d4a5a-b39f-479a-99f1-50c2ad93a36a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:57.643658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3d4a5a-b39f-479a-99f1-50c2ad93a36a None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.644173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3d4a5a-b39f-479a-99f1-50c2ad93a36a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.644724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3d4a5a-b39f-479a-99f1-50c2ad93a36a None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.645612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3d4a5a-b39f-479a-99f1-50c2ad93a36a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:57.647419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3d4a5a-b39f-479a-99f1-50c2ad93a36a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:57.683291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 173/350] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:57 2022] POST /identity/v3/users => generated 248 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:57.699457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:57.732559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be897017-cf99-4fdb-a482-14fd6a0e707b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:57.735247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3d4a5a-b39f-479a-99f1-50c2ad93a36a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:57.782812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:57.783472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.783749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.784020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:57.798539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:57.799995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:37:57.820217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:57.821383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:37:57.822768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:37:57.822768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:37:57.822768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:57.827224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 178/351] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:57.828840 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:57.837596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:57.938288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3d4a5a-b39f-479a-99f1-50c2ad93a36a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:57.997959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91870c1-6a9b-461f-853a-33e34a5e79fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:57.998622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91870c1-6a9b-461f-853a-33e34a5e79fd None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:57.998905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91870c1-6a9b-461f-853a-33e34a5e79fd None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:57.999235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91870c1-6a9b-461f-853a-33e34a5e79fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:58.003227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9005b9e-94ee-46b6-a175-b9f65121e8f0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:58.013587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91870c1-6a9b-461f-853a-33e34a5e79fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:58.015167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91870c1-6a9b-461f-853a-33e34a5e79fd None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-c91870c1-6a9b-461f-853a-33e34a5e79fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:37:58.035647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:58.036710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:37:58.037560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:37:58.037560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:37:58.037560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:58.037560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 174/352] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:37:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:37:58.038175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93dac6-fce6-48a2-836a-65c1c021a36f None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:58.040742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93dac6-fce6-48a2-836a-65c1c021a36f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:58.040742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93dac6-fce6-48a2-836a-65c1c021a36f None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:58.041691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 175/353] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:37:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:37:58.045034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b93dac6-fce6-48a2-836a-65c1c021a36f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:58.059619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4454c583-011b-4d45-b671-da4225c3a737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:58.060199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4454c583-011b-4d45-b671-da4225c3a737 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:58.060480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4454c583-011b-4d45-b671-da4225c3a737 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:58.060749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4454c583-011b-4d45-b671-da4225c3a737 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:58.061412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4454c583-011b-4d45-b671-da4225c3a737 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:58.061713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4454c583-011b-4d45-b671-da4225c3a737 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:58.063322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4454c583-011b-4d45-b671-da4225c3a737 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:58.072124 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:58.081056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 179/354] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:37:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:37:58.082666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24afcf64-ea93-4f00-b47a-6b9476bcad26 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:58.083134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24afcf64-ea93-4f00-b47a-6b9476bcad26 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:58.083600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24afcf64-ea93-4f00-b47a-6b9476bcad26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:58.107921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-24afcf64-ea93-4f00-b47a-6b9476bcad26 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:37:58.109500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-24afcf64-ea93-4f00-b47a-6b9476bcad26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:37:58.175396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:58.193100 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b93dac6-fce6-48a2-836a-65c1c021a36f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:58.221123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 180/355] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:37:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:37:58.227852 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24afcf64-ea93-4f00-b47a-6b9476bcad26 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:58.246844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:58.248225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:58.248225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:58.248487 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:58.249191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:58.249506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:58.256513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:58.272164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 176/356] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:37:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:37:58.275149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:58.451256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8bbcf-980d-4ba1-82f9-149e2273ff8e None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:58.475359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24afcf64-ea93-4f00-b47a-6b9476bcad26 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:58.508920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f803be-6552-4dc4-8ebb-56362e8a6889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:58.509531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f803be-6552-4dc4-8ebb-56362e8a6889 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:58.510800 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f803be-6552-4dc4-8ebb-56362e8a6889 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:58.511280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f803be-6552-4dc4-8ebb-56362e8a6889 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:58.512073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f803be-6552-4dc4-8ebb-56362e8a6889 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:58.513631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f803be-6552-4dc4-8ebb-56362e8a6889 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:58.536480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:58.546792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 177/357] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:58 2022] POST /identity/v3/users => generated 249 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:37:58.555014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b14a63b-3839-4cd5-b680-46d18ee1954a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:58.555637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14a63b-3839-4cd5-b680-46d18ee1954a None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:58.555913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14a63b-3839-4cd5-b680-46d18ee1954a None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:58.556197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14a63b-3839-4cd5-b680-46d18ee1954a None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:58.556907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b14a63b-3839-4cd5-b680-46d18ee1954a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:58.558917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b14a63b-3839-4cd5-b680-46d18ee1954a None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:58.586889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 181/358] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:37:58 2022] POST /identity/v3/users => generated 248 bytes in 360 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:37:58.691527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f803be-6552-4dc4-8ebb-56362e8a6889 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:58.710414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:58.841865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f803be-6552-4dc4-8ebb-56362e8a6889 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:58.886407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:58.887112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:58.887403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:58.887682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:58.888413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:58.889860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:58.898382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.sql.core [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 24 08:37:58.898382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 24 08:37:58.898382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [parameters: {'id': '401bea8763d94d1e96a0e3c8d9586ad9', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 24 08:37:58.898382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=93079) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result.read() Oct 24 08:37:58.907133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return ref.to_dict() Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:37:58.908076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application yield resource Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.session.rollback() Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.force_reraise() Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise self.value Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application session.commit() Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 24 08:37:58.909062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._prepare_impl() Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.session.flush() Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._flush(objects) Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application compat.raise_( Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application flush_context.execute() Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rec.execute(self) Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 24 08:37:58.909929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:37:58.910826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result.read() Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:37:58.911643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [parameters: {'id': '401bea8763d94d1e96a0e3c8d9586ad9', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:58.913917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 24 08:37:58.915999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:37:58.921895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 178/359] 10.222.0.39 () {60 vars in 1131 bytes} [Mon Oct 24 08:37:58 2022] POST /identity/v3/roles => generated 144 bytes in 218 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:58.924698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b14a63b-3839-4cd5-b680-46d18ee1954a None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:59.059226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b14a63b-3839-4cd5-b680-46d18ee1954a None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:59.099237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567dc472-f921-4d5a-8568-bc9671021a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:59.099924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567dc472-f921-4d5a-8568-bc9671021a9b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:59.100231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567dc472-f921-4d5a-8568-bc9671021a9b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:59.100502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567dc472-f921-4d5a-8568-bc9671021a9b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:59.113823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567dc472-f921-4d5a-8568-bc9671021a9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-567dc472-f921-4d5a-8568-bc9671021a9b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:37:59.120095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:37:59.121147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:37:59.122436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 182/360] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:37:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:37:59.130973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:37:59.258722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:37:59.292821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221a8bea-aa41-4299-8a8b-f4f27fefef8e None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:37:59.343303 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b61a9e7-7143-4202-8d18-abbaaa82e6da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:37:59.343925 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b61a9e7-7143-4202-8d18-abbaaa82e6da None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:37:59.344306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b61a9e7-7143-4202-8d18-abbaaa82e6da None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:37:59.344583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b61a9e7-7143-4202-8d18-abbaaa82e6da None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:37:59.345259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b61a9e7-7143-4202-8d18-abbaaa82e6da None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:37:59.345568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b61a9e7-7143-4202-8d18-abbaaa82e6da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:37:59.346984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b61a9e7-7143-4202-8d18-abbaaa82e6da None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:37:59.356491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 179/361] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:37:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:00.252900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1792db-e926-48b6-955c-6afdd01d6a9f None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:00.253938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1792db-e926-48b6-955c-6afdd01d6a9f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:00.254433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1792db-e926-48b6-955c-6afdd01d6a9f None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:00.256307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 183/362] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:00.262076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68931690-9f3f-46de-a3cb-cafdcdd8a98e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:00.262959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68931690-9f3f-46de-a3cb-cafdcdd8a98e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:00.263465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68931690-9f3f-46de-a3cb-cafdcdd8a98e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:00.283509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-68931690-9f3f-46de-a3cb-cafdcdd8a98e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:00.285092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-68931690-9f3f-46de-a3cb-cafdcdd8a98e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:00.368049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 180/363] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:00.539787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0516e-ad4f-4605-928b-b2ad13e41e3a None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:00.540411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0516e-ad4f-4605-928b-b2ad13e41e3a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:00.540894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0516e-ad4f-4605-928b-b2ad13e41e3a None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:00.543327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 184/364] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:00.552981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2953135-508a-4484-9ebe-35e99fd45b4e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:00.553569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2953135-508a-4484-9ebe-35e99fd45b4e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:00.554073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2953135-508a-4484-9ebe-35e99fd45b4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:00.584823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c2953135-508a-4484-9ebe-35e99fd45b4e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:00.586390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c2953135-508a-4484-9ebe-35e99fd45b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:00.662740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:00.671048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 181/365] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:00.790316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb0516e-ad4f-4605-928b-b2ad13e41e3a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:00.853322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:00.930997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb0516e-ad4f-4605-928b-b2ad13e41e3a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:00.951267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2953135-508a-4484-9ebe-35e99fd45b4e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.011762 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39877f6d-6c29-4837-94ca-62a4010dbd86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:01.012431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39877f6d-6c29-4837-94ca-62a4010dbd86 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.012711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39877f6d-6c29-4837-94ca-62a4010dbd86 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.013527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39877f6d-6c29-4837-94ca-62a4010dbd86 None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.025958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39877f6d-6c29-4837-94ca-62a4010dbd86 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-39877f6d-6c29-4837-94ca-62a4010dbd86 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:01.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:01.033390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:01.034193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 185/366] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:01.043560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39877f6d-6c29-4837-94ca-62a4010dbd86 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.170265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2953135-508a-4484-9ebe-35e99fd45b4e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:01.193763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39877f6d-6c29-4837-94ca-62a4010dbd86 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:01.196468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:01.211977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4448742-1a00-445c-8d4e-2b8cb8a85267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:01.212650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4448742-1a00-445c-8d4e-2b8cb8a85267 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.213456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4448742-1a00-445c-8d4e-2b8cb8a85267 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.213725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4448742-1a00-445c-8d4e-2b8cb8a85267 None admin] PATH_INFO: `/v3/roles/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.224960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4448742-1a00-445c-8d4e-2b8cb8a85267 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-d4448742-1a00-445c-8d4e-2b8cb8a85267 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:01.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:01.231692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:01.233101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 182/367] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:01.239294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:01.239590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4448742-1a00-445c-8d4e-2b8cb8a85267 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.239910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.240218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.240593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.241258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] RBAC: Adding query filter params (name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:01.241553 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:01.243224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:01.254237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 186/368] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:01.262264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.406736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e320a9d0-c4d4-4ba8-9d69-f0fec578c2ba None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:01.408678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:01.443235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:01.443878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.444178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.444545 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.458428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:01.460341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:01.477761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:01.478801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:01.479657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:01.479657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:01.479657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:01.479657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 187/369] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:01.492738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.501681 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4448742-1a00-445c-8d4e-2b8cb8a85267 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:01.528581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:01.537714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:01.538638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.539152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.539628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.540527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] RBAC: Adding query filter params (name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:01.541582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:01.543153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:01.552740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 183/370] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:01.559185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.615327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91031b0-0a1d-4139-bcaf-00139a611ca7 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:01.653774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8ef58b-3cc9-4d53-a822-93d6b725dacd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:01.654377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ef58b-3cc9-4d53-a822-93d6b725dacd None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.654653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ef58b-3cc9-4d53-a822-93d6b725dacd None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.654923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ef58b-3cc9-4d53-a822-93d6b725dacd None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.655559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8ef58b-3cc9-4d53-a822-93d6b725dacd None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:01.655858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8ef58b-3cc9-4d53-a822-93d6b725dacd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:01.657609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8ef58b-3cc9-4d53-a822-93d6b725dacd None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:01.673028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 188/371] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:01.674440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8031a3-861e-41f5-9562-c275c8d7effb None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.674621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8031a3-861e-41f5-9562-c275c8d7effb None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.674880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8031a3-861e-41f5-9562-c275c8d7effb None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.676194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 189/372] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:01.681167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8031a3-861e-41f5-9562-c275c8d7effb None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.687379 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e420b587-8c4e-4f9d-80df-ed35038406ab None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:01.708618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:01.725581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb9e5d8-c018-4d7a-9524-9696309439f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:01.726560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb9e5d8-c018-4d7a-9524-9696309439f2 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.726842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb9e5d8-c018-4d7a-9524-9696309439f2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.727117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb9e5d8-c018-4d7a-9524-9696309439f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.740059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb9e5d8-c018-4d7a-9524-9696309439f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:01.741561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb9e5d8-c018-4d7a-9524-9696309439f2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-9eb9e5d8-c018-4d7a-9524-9696309439f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:01.756634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:01.757804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:01.759262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:01.759262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:01.759262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:01.759262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 184/373] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:01.763578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065d422-a15f-468e-aea4-5853fd0674f3 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.763869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065d422-a15f-468e-aea4-5853fd0674f3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.764164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065d422-a15f-468e-aea4-5853fd0674f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.768689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:01.783749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c065d422-a15f-468e-aea4-5853fd0674f3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:01.784988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c065d422-a15f-468e-aea4-5853fd0674f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:01.798924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8031a3-861e-41f5-9562-c275c8d7effb None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:01.835406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:01.836048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.836355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.836626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] PATH_INFO: `/v3/users/nova` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.847528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:01.849034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:01.859107 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 185/374] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:01.863181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:01.864062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 190/375] 10.222.0.39 () {56 vars in 1093 bytes} [Mon Oct 24 08:38:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:01.865703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.870014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bb40ba-0520-49f8-9bb2-af0fc398c535 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:01.870308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bb40ba-0520-49f8-9bb2-af0fc398c535 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:01.870649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bb40ba-0520-49f8-9bb2-af0fc398c535 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:01.877017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 186/376] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:01 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:01.877017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bb40ba-0520-49f8-9bb2-af0fc398c535 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:01.989321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d65e4-ab95-45e6-ade1-8ce4a5272767 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.017513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bb40ba-0520-49f8-9bb2-af0fc398c535 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.019311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:02.029837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f929691-ecbc-47b6-9f5c-e8074cf07460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.030593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f929691-ecbc-47b6-9f5c-e8074cf07460 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.030976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f929691-ecbc-47b6-9f5c-e8074cf07460 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.031343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f929691-ecbc-47b6-9f5c-e8074cf07460 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.032089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f929691-ecbc-47b6-9f5c-e8074cf07460 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:02.032532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f929691-ecbc-47b6-9f5c-e8074cf07460 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:02.034078 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f929691-ecbc-47b6-9f5c-e8074cf07460 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:02.042493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:02.048609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 191/377] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:02.053595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc441e9-a906-49b6-bf2c-7a9f28eb79e9 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.054905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc441e9-a906-49b6-bf2c-7a9f28eb79e9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.056029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc441e9-a906-49b6-bf2c-7a9f28eb79e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.070615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.071564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.072007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.073142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.074105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:02.074597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:02.076278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:02.079619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-bfc441e9-a906-49b6-bf2c-7a9f28eb79e9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:02.080961 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bfc441e9-a906-49b6-bf2c-7a9f28eb79e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:02.119764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 187/378] 10.222.0.39 () {56 vars in 1133 bytes} [Mon Oct 24 08:38:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:02.121086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.161258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 192/379] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:02.161258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc441e9-a906-49b6-bf2c-7a9f28eb79e9 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.241371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd76352b-875e-48b0-b49f-cf640cbf8208 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.273563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc441e9-a906-49b6-bf2c-7a9f28eb79e9 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.292627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.293854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.294511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.294937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] PATH_INFO: `/v3/users/glance` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.305877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:02.308115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:02.309480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:02.318667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.319525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.319974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.320301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:02.329510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:02.330641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 24 08:38:02.331565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:02.334634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:02.334912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 188/380] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/users/glance => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:02.336403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.342945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:02.362477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:02.363458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:02.364381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:02.364381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:02.364381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:02.364381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 193/381] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:02.364608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.368261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:02.468792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b281aea9-3c8a-45fb-805d-51c8b563d16a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.487414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc2882-5102-45ff-850e-e8e8a55df736 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.506563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3349e36-ae11-4ef3-9293-94cf821dc0c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.507113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3349e36-ae11-4ef3-9293-94cf821dc0c7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.507393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3349e36-ae11-4ef3-9293-94cf821dc0c7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.507675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3349e36-ae11-4ef3-9293-94cf821dc0c7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.519171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3349e36-ae11-4ef3-9293-94cf821dc0c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:02.524115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.524763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.525043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.525849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-e3349e36-ae11-4ef3-9293-94cf821dc0c7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:02.526368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:02.532560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:02.533680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:02.534478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:02.534478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:02.534632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 189/382] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:02.534632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3349e36-ae11-4ef3-9293-94cf821dc0c7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.534777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:02.574579 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 194/383] 10.222.0.39 () {56 vars in 1137 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:02.575566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.615689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:02.619579 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:02.638163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3349e36-ae11-4ef3-9293-94cf821dc0c7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.674441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2c8ce1-c79a-4343-9e26-6dcbb233e9d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.675065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2c8ce1-c79a-4343-9e26-6dcbb233e9d5 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.675346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2c8ce1-c79a-4343-9e26-6dcbb233e9d5 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.675623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2c8ce1-c79a-4343-9e26-6dcbb233e9d5 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.676311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2c8ce1-c79a-4343-9e26-6dcbb233e9d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:02.676614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2c8ce1-c79a-4343-9e26-6dcbb233e9d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:02.678035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2c8ce1-c79a-4343-9e26-6dcbb233e9d5 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:02.685286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b222216f-7d08-4317-ab23-947f86fb319d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.691476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 190/384] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:02.692852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f9a87d-3d35-458b-a372-ffd633df17c2 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.693140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f9a87d-3d35-458b-a372-ffd633df17c2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.693411 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f9a87d-3d35-458b-a372-ffd633df17c2 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.694642 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 191/385] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:02.695340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f9a87d-3d35-458b-a372-ffd633df17c2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.724929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7d15d4-46a8-477b-a87d-80fcd1f8fb6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.725603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d15d4-46a8-477b-a87d-80fcd1f8fb6c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.725920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d15d4-46a8-477b-a87d-80fcd1f8fb6c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.726279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d15d4-46a8-477b-a87d-80fcd1f8fb6c None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.742604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7d15d4-46a8-477b-a87d-80fcd1f8fb6c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-5d7d15d4-46a8-477b-a87d-80fcd1f8fb6c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:02.748256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:02.749186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:02.749971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 195/386] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:02.750636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7d15d4-46a8-477b-a87d-80fcd1f8fb6c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.815132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f9a87d-3d35-458b-a372-ffd633df17c2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.857969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.858587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.858859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.859129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.865571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:02.870604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] RBAC: Adding query filter params (name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:02.871070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7d15d4-46a8-477b-a87d-80fcd1f8fb6c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:02.871283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:02.873926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:02.892463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 192/387] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:02.892463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:02.900632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:02.937916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe86af0b-a522-47c9-887e-d200c9c00a0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:02.938556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86af0b-a522-47c9-887e-d200c9c00a0f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.938835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86af0b-a522-47c9-887e-d200c9c00a0f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.939107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86af0b-a522-47c9-887e-d200c9c00a0f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:02.953676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe86af0b-a522-47c9-887e-d200c9c00a0f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:02.955151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe86af0b-a522-47c9-887e-d200c9c00a0f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-fe86af0b-a522-47c9-887e-d200c9c00a0f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:02.982233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:02.983743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:02.989198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:02.989198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:02.989198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:02.989198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 196/388] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:02.989198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991f80d-521d-434e-a0de-d34a3b24f84f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:02.989198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991f80d-521d-434e-a0de-d34a3b24f84f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:02.989198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991f80d-521d-434e-a0de-d34a3b24f84f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.007335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5991f80d-521d-434e-a0de-d34a3b24f84f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:03.008570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5991f80d-521d-434e-a0de-d34a3b24f84f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:03.040046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fb4424-e45f-453d-871d-c7ce0e0a6b8b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.073865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.074407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.074681 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.074950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.078805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 197/389] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:03.079737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5991f80d-521d-434e-a0de-d34a3b24f84f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.081491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.082936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:03.095158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:03.096281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:03.097400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:03.097400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:03.097400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:03.097400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:03.097400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:03.097400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:03.097400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:03.097400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:03.100472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 193/390] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:03.102115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.162614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:03.171644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:03.195019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5991f80d-521d-434e-a0de-d34a3b24f84f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.205242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebbc523-6f5c-4e64-82ce-63e4ab9cb228 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.237479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.238105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.238375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.238647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.239318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] RBAC: Adding query filter params (name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:03.239611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.241022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.242732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.243769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.244788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.245175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.254920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 198/391] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:03.254920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.256462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.257998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:03.273225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:03.274355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:03.275467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:03.275467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:03.275467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:03.275467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 194/392] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:03.275467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.369952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d90e3f-039b-4d2d-b66b-e75203aae37f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.382765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae514fb-19fe-4330-832b-9d7443cf5b53 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.396838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:03.404907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.405513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.405791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.406064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.406696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:03.406996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.408394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.417307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.417896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.418175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.418455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.419136 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:03.419443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.420641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 199/393] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:03.420904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.421477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.424221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:03.439073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 195/394] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:03.439904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.529151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652b04d1-241f-4fc0-8625-942f37c2421d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.541847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c379384d-0488-4bfe-b7c5-263cf642a4ae None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.564634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.565256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.565535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.565804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.575588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.576213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.576323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.576496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.576769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.577396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:03.577666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.577755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.579048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.590742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 196/395] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:03.591594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:03.594102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:03.595141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:03.596812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:03.596812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:03.596812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:03.596812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 200/396] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:03.596812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.629335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:03.655440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:03.698087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0360920d-c019-4031-80c3-294e3ce4856c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.700932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fecdebb-70d5-49f9-91dd-1a82501723c2 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.731262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.731884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.732185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.732456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.734834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.735520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.735813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.736076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.742626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.743637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.744412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.745089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:03.756721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:03.757718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:03.758803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:03.758803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:03.758803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:03.758803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:03.758803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:03.758803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:03.758803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:03.758803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:03.759298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:03.760760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:03.761634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:03.761634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:03.761634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:03.761634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 197/397] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:03.761634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.763252 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 201/398] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:03.764078 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.847063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:03.862403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e5e0fe-4f54-4b56-8703-78f514aaee04 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.862834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03f0d1b-6122-4e1b-8618-fdc96ac640d6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:03.877546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:03.897687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c880ee-45ec-4774-9da0-70cb06221162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.898016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:03.898271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c880ee-45ec-4774-9da0-70cb06221162 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.898554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c880ee-45ec-4774-9da0-70cb06221162 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.898645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:03.898830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c880ee-45ec-4774-9da0-70cb06221162 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/4244858dfb464d029b6f6bed0491557d/roles/a1265c18dedb42d5943905f144e48fb7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.898921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:03.899144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:03.907474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.908925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.913118 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c880ee-45ec-4774-9da0-70cb06221162 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=4244858dfb464d029b6f6bed0491557d, role_id=a1265c18dedb42d5943905f144e48fb7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:03.914557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c880ee-45ec-4774-9da0-70cb06221162 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:03.921166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:03.922271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 202/399] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/users/neutron => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:03.923172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:03.940596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 198/400] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:03 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/4244858dfb464d029b6f6bed0491557d/roles/a1265c18dedb42d5943905f144e48fb7 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:03.940596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c880ee-45ec-4774-9da0-70cb06221162 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.047500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d1696d-f6c4-47eb-945b-303e637aff3e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.078481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c880ee-45ec-4774-9da0-70cb06221162 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.089168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.089945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.090290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.091022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.091758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:04.092487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.093951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.108460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 203/401] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:04.109590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.121734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.122327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.122600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.123072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.123660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:04.123961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.125590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.128795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:04.140177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 199/402] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:04.140909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.155562 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:04.230293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69901de5-eee6-4f6a-bc0e-a99d2160b685 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.254937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfbe084-f824-47e4-87b4-ef1e21e302eb None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.269437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.270123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.270437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.270759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.271443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:04.271788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.273435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.286849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 204/403] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:04.289187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.289757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.290345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.290613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.290886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.292676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:04.292984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.294416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.312451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 200/404] 10.222.0.39 () {56 vars in 1139 bytes} [Mon Oct 24 08:38:04 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:04.313260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.370046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:04.399984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:04.409444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1cc983-8a83-46b7-ab6f-2de776b65a57 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.422588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6511c729-edbd-44f7-a7d3-99e146544cf2 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.445978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.446600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.446873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.447136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.455581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.457058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.460172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcefd2e-118d-41f5-8560-a0bcea899a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.460889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcefd2e-118d-41f5-8560-a0bcea899a4a None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.461193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcefd2e-118d-41f5-8560-a0bcea899a4a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.461468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcefd2e-118d-41f5-8560-a0bcea899a4a None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/5a4fd11d5d85479c9aa1d87b23a13141/roles/a1265c18dedb42d5943905f144e48fb7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:04.468542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:04.469494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:04.470390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 205/405] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:04 2022] GET /identity/v3/users/cinder => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:04.470981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.484279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcefd2e-118d-41f5-8560-a0bcea899a4a None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=5a4fd11d5d85479c9aa1d87b23a13141, role_id=a1265c18dedb42d5943905f144e48fb7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.485936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcefd2e-118d-41f5-8560-a0bcea899a4a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.508376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 201/406] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:04 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/5a4fd11d5d85479c9aa1d87b23a13141/roles/a1265c18dedb42d5943905f144e48fb7 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:04.508376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcefd2e-118d-41f5-8560-a0bcea899a4a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.612529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21abe8e-9f89-4b85-91f1-781601d3705b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.651736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcefd2e-118d-41f5-8560-a0bcea899a4a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.654690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:04.664012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc33bb-b31e-42e5-a1e1-ae4b4e146787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.664702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc33bb-b31e-42e5-a1e1-ae4b4e146787 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.665052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc33bb-b31e-42e5-a1e1-ae4b4e146787 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.665909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc33bb-b31e-42e5-a1e1-ae4b4e146787 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.667187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dc33bb-b31e-42e5-a1e1-ae4b4e146787 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.668906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dc33bb-b31e-42e5-a1e1-ae4b4e146787 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.699771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 206/407] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:38:04 2022] POST /identity/v3/users => generated 247 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:04.700581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc33bb-b31e-42e5-a1e1-ae4b4e146787 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.700688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.701385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.701664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.701936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.719233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.720846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:04.721406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:04.740799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:04.741773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:04.742625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:04.742625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:04.742625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:04.745796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 202/408] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:04.753243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.844208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc33bb-b31e-42e5-a1e1-ae4b4e146787 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.890997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.891765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.892107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.892456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.894023 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:04.894395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.895899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:04.922250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e228e5-c08a-4719-ba99-0302a43c8a7c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:04.922753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 207/409] 10.222.0.39 () {56 vars in 1137 bytes} [Mon Oct 24 08:38:04 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:04.925287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:04.981178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:04.981902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:04.982243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:04.982579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:04.983278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:04.983644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:04.985095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:05.000397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 203/410] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:05.006118 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:05.023421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:05.032254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:05.071761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a804741-e2d1-4148-8228-2261670b15a0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:05.125031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:05.125597 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:05.125967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:05.126372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:05.139342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:05.141565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:05.147299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a92091-5ff2-4b23-96d1-5d7c6ea41894 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:05.160950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:05.162125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:05.163216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:05.163216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:05.163216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:05.169234 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 208/411] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:05.171083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:05.193167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:05.194219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:05.194597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:05.194964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:05.205001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:05.207396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:05.222315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:05.223414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:05.224485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:05.224485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:05.224485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:05.224485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:05.224485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:05.224485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:05.224485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:05.224485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:05.226531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 204/412] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:05.237287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:05.362368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bad7b0-5e3b-4310-be5c-2bc3e99116f8 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:05.398894 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:05.402757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:05.430509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:05.431519 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:05.433219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:05.433647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:05.434467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:05.434910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:05.436490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:05.451280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3350dfa-a81a-48ee-accb-87d941b1896d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:05.451772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 209/413] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:05.457191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:05.497117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:05.497970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:05.498087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:05.498332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:05.499125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:05.499341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:05.500867 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:05.523348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 205/414] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:05.530016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:05.611843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3703da87-4044-4872-89be-14766941e1d6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:05.666603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dc3b4d-c7f1-4506-8796-49f3e0fcb443 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:05.698829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:05.699304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:05.699614 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:05.699869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:05.707725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:05.709125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:05.723277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:05.724230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:05.725088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 210/415] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:05.731245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f99e4-f823-4e8e-af69-ac88bef01415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:05.732492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:05.736288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f99e4-f823-4e8e-af69-ac88bef01415 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:05.737552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f99e4-f823-4e8e-af69-ac88bef01415 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:05.737965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f99e4-f823-4e8e-af69-ac88bef01415 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/8bfae94e0fa041c9802729370a810245/roles/a1265c18dedb42d5943905f144e48fb7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:05.750638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:05.770505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034f99e4-f823-4e8e-af69-ac88bef01415 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=8bfae94e0fa041c9802729370a810245, role_id=a1265c18dedb42d5943905f144e48fb7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:05.771977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034f99e4-f823-4e8e-af69-ac88bef01415 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:05.808536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:05.823615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 206/416] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:05 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/8bfae94e0fa041c9802729370a810245/roles/a1265c18dedb42d5943905f144e48fb7 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:05.911777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e60e05-f75e-4cdb-ad2c-e08fb1a95a56 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:05.964088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2ef846-ad40-42e2-8577-8017cdf3a280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:05.965001 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2ef846-ad40-42e2-8577-8017cdf3a280 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:05.965553 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2ef846-ad40-42e2-8577-8017cdf3a280 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:05.966144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2ef846-ad40-42e2-8577-8017cdf3a280 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:05.967231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2ef846-ad40-42e2-8577-8017cdf3a280 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:05.967647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2ef846-ad40-42e2-8577-8017cdf3a280 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:05.969511 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2ef846-ad40-42e2-8577-8017cdf3a280 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:06.003189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 211/417] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:06.004703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f99e4-f823-4e8e-af69-ac88bef01415 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:06.181445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f99e4-f823-4e8e-af69-ac88bef01415 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:06.219828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e983e4-5e40-4639-b825-9e49dfcfe000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:06.220448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e983e4-5e40-4639-b825-9e49dfcfe000 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:06.220731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e983e4-5e40-4639-b825-9e49dfcfe000 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:06.221001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e983e4-5e40-4639-b825-9e49dfcfe000 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/412f520f90564e44819e8bf947992efb/roles/a1265c18dedb42d5943905f144e48fb7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:06.239175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e983e4-5e40-4639-b825-9e49dfcfe000 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=412f520f90564e44819e8bf947992efb, role_id=a1265c18dedb42d5943905f144e48fb7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:06.240722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e983e4-5e40-4639-b825-9e49dfcfe000 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:06.265240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 207/418] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:06 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/412f520f90564e44819e8bf947992efb/roles/a1265c18dedb42d5943905f144e48fb7 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:06.430496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e81c3cc-1059-4c54-8a5e-c583df7a2080 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:06.431019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e81c3cc-1059-4c54-8a5e-c583df7a2080 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:06.431019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e81c3cc-1059-4c54-8a5e-c583df7a2080 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:06.433424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 212/419] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:06.440316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218c5a72-7101-4383-898d-7dae286614d2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:06.440945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218c5a72-7101-4383-898d-7dae286614d2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:06.441480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218c5a72-7101-4383-898d-7dae286614d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:06.473537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-218c5a72-7101-4383-898d-7dae286614d2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:06.475132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-218c5a72-7101-4383-898d-7dae286614d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:06.521790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:06.588009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 208/420] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:06.913260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e81c3cc-1059-4c54-8a5e-c583df7a2080 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:07.074351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:07.082735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e81c3cc-1059-4c54-8a5e-c583df7a2080 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:07.134395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2648e8-0fdc-427d-aed3-b3d9c1324934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:07.135694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2648e8-0fdc-427d-aed3-b3d9c1324934 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:07.135975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2648e8-0fdc-427d-aed3-b3d9c1324934 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:07.136276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2648e8-0fdc-427d-aed3-b3d9c1324934 None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:07.155470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2648e8-0fdc-427d-aed3-b3d9c1324934 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-8c2648e8-0fdc-427d-aed3-b3d9c1324934 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:07.160607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:07.161535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:07.162353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 213/421] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:07.170438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218c5a72-7101-4383-898d-7dae286614d2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:07.334782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218c5a72-7101-4383-898d-7dae286614d2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:07.389709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:07.390336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:07.390617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:07.390891 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:07.391566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] RBAC: Adding query filter params (name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:07.391873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:07.393394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:07.406492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 209/422] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:07.412447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2648e8-0fdc-427d-aed3-b3d9c1324934 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:07.555148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2648e8-0fdc-427d-aed3-b3d9c1324934 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:07.603863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b40646d-b9ee-4ecc-bc69-a1f3aa286807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:07.604612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40646d-b9ee-4ecc-bc69-a1f3aa286807 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:07.604904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40646d-b9ee-4ecc-bc69-a1f3aa286807 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:07.605267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40646d-b9ee-4ecc-bc69-a1f3aa286807 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:07.618321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b40646d-b9ee-4ecc-bc69-a1f3aa286807 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:07.619955 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b40646d-b9ee-4ecc-bc69-a1f3aa286807 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-0b40646d-b9ee-4ecc-bc69-a1f3aa286807 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:07.638390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:07.639371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:07.640261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:07.640261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:07.640261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:07.640261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 214/423] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:07.645953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:07.705711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:07.809103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de2130e-2cf5-4a84-9bb0-d71e4483754c None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:07.809406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de2130e-2cf5-4a84-9bb0-d71e4483754c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:07.809618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de2130e-2cf5-4a84-9bb0-d71e4483754c None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:07.810777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e6d18-f8e6-4f56-b7c9-fda79bef98a4 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:07.811207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 215/424] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:07.817278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d7161-d7c3-4220-b0de-5835ee4c2339 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:07.817629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d7161-d7c3-4220-b0de-5835ee4c2339 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:07.817980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d7161-d7c3-4220-b0de-5835ee4c2339 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:07.857398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a39d7161-d7c3-4220-b0de-5835ee4c2339 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:07.858751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a39d7161-d7c3-4220-b0de-5835ee4c2339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:07.860483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:07.861636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:07.862137 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:07.862574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:07.863413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:07.863882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:07.866754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:07.872539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:07.884648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 210/425] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:07.890923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:08.001861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 216/426] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:08.003456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f2fc4-5a35-48f1-8d17-3705e5984ff2 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:08.004057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f2fc4-5a35-48f1-8d17-3705e5984ff2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:08.004535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f2fc4-5a35-48f1-8d17-3705e5984ff2 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:08.006437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 217/427] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:08.012315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f527ab-e028-4d13-92ea-1ac90fdcf3b7 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:08.012982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f527ab-e028-4d13-92ea-1ac90fdcf3b7 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:08.013462 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f527ab-e028-4d13-92ea-1ac90fdcf3b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:08.034852 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-65f527ab-e028-4d13-92ea-1ac90fdcf3b7 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:08.036539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-65f527ab-e028-4d13-92ea-1ac90fdcf3b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:08.047440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:08.077348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc404c82-8d33-4900-9d45-14d91caf7a4f None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:08.121496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 218/428] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:08.159284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:08.159929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:08.160228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:08.160501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] PATH_INFO: `/v3/users/nova` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:08.172632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:08.174114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:08.193120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:08.194109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 211/429] 10.222.0.39 () {56 vars in 1093 bytes} [Mon Oct 24 08:38:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:08.201130 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f527ab-e028-4d13-92ea-1ac90fdcf3b7 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:08.314903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:08.362221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f527ab-e028-4d13-92ea-1ac90fdcf3b7 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:08.406578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:08.407563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:08.408054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:08.408577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:08.409501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:08.409998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:08.411646 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:08.433504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 219/430] 10.222.0.39 () {56 vars in 1133 bytes} [Mon Oct 24 08:38:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:08.439587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:08.440475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:08.605210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6c230e-33cb-496a-a43a-f73193365c46 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:08.622957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bdadd9-9312-42d4-a6ce-606494162896 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:08.652157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:08.660526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb251d9d-a907-47b5-bf23-c065b268272d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:08.661880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb251d9d-a907-47b5-bf23-c065b268272d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:08.662390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb251d9d-a907-47b5-bf23-c065b268272d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:08.662859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb251d9d-a907-47b5-bf23-c065b268272d None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:08.677367 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb251d9d-a907-47b5-bf23-c065b268272d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-fb251d9d-a907-47b5-bf23-c065b268272d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:08.684917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:08.685871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:08.686758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:08.689587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 220/431] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:08.690948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb251d9d-a907-47b5-bf23-c065b268272d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:08.695497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67aadf1-65d7-444b-b9cd-2a77e8042261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:08.698643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67aadf1-65d7-444b-b9cd-2a77e8042261 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:08.699042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67aadf1-65d7-444b-b9cd-2a77e8042261 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:08.699416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67aadf1-65d7-444b-b9cd-2a77e8042261 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:08.727397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:08.821868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67aadf1-65d7-444b-b9cd-2a77e8042261 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:08.823913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67aadf1-65d7-444b-b9cd-2a77e8042261 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:08.852109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb251d9d-a907-47b5-bf23-c065b268272d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:08.926932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88e4355-c07f-49e6-a312-97b1000abbd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:08.927977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88e4355-c07f-49e6-a312-97b1000abbd9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:08.928418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88e4355-c07f-49e6-a312-97b1000abbd9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:08.928890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88e4355-c07f-49e6-a312-97b1000abbd9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:08.949639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88e4355-c07f-49e6-a312-97b1000abbd9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-e88e4355-c07f-49e6-a312-97b1000abbd9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:08.955872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:08.957827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:08.959634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:08.964726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 221/432] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:08.965892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88e4355-c07f-49e6-a312-97b1000abbd9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:09.000173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 212/433] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 561 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 24 08:38:09.001542 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b22a92-bd64-483e-9ce8-93f941db1222 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:09.001828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b22a92-bd64-483e-9ce8-93f941db1222 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:09.002189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b22a92-bd64-483e-9ce8-93f941db1222 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:09.008350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 213/434] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:09.008350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b22a92-bd64-483e-9ce8-93f941db1222 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:09.052636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:09.102333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:09.117656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88e4355-c07f-49e6-a312-97b1000abbd9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:09.162612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:09.163387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:09.163802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:09.164306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:09.165300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] RBAC: Adding query filter params (name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:09.165879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:09.167435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:09.184082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 222/435] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:09.184082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:09.215650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b22a92-bd64-483e-9ce8-93f941db1222 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:09.273857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371ed8b3-fba3-4c78-84e8-2ace4ab057a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:09.274649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371ed8b3-fba3-4c78-84e8-2ace4ab057a6 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:09.275024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371ed8b3-fba3-4c78-84e8-2ace4ab057a6 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:09.276091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371ed8b3-fba3-4c78-84e8-2ace4ab057a6 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:09.276815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371ed8b3-fba3-4c78-84e8-2ace4ab057a6 None admin] RBAC: Adding query filter params (name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:09.277355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371ed8b3-fba3-4c78-84e8-2ace4ab057a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:09.278737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371ed8b3-fba3-4c78-84e8-2ace4ab057a6 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:09.289095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 214/436] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:09.295263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4933b-c4a6-4d87-8380-3e721babcc0a None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:09.295834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4933b-c4a6-4d87-8380-3e721babcc0a None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:09.296572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4933b-c4a6-4d87-8380-3e721babcc0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:09.331043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b3e4933b-c4a6-4d87-8380-3e721babcc0a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:09.333341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b3e4933b-c4a6-4d87-8380-3e721babcc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:09.357295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4c5d58-14b1-4da8-aa84-967ac316642d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:09.386998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:09.425968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:09.427006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:09.427495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:09.427970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:09.450829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:09.453765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:09.460913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 215/437] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:09.465158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e4933b-c4a6-4d87-8380-3e721babcc0a None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:09.485111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:09.486085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:09.486946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:09.486946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:09.486946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:09.491603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 223/438] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:09.491603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:09.523902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:09.645733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01980639-2ebf-4953-8dbb-6b22b7beed97 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:09.661056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e4933b-c4a6-4d87-8380-3e721babcc0a None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:09.714314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:09.716502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:09.717375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:09.718007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:09.735139 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:09.736502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:09.737171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:09.737785 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:09.747692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:09.750601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:09.761826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:09.765365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:09.780875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:09.782773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:09.784602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:09.784602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:09.784602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:09.787782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 224/439] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:09.789677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:09.814471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:09.816307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:09.818073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:09.818073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:09.818073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:09.818073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 216/440] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:09.819846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:09.834482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:09.955648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37235a18-cec1-4f65-b203-13bb97ceedf0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:09.993493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:10.008734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9849d86-2c0d-4685-8d7d-4d857d1f688b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:10.009357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5ebaa6-2ffc-4802-a732-01c4885f8cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.010189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ebaa6-2ffc-4802-a732-01c4885f8cbe None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.010578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ebaa6-2ffc-4802-a732-01c4885f8cbe None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.011209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ebaa6-2ffc-4802-a732-01c4885f8cbe None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.011580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ebaa6-2ffc-4802-a732-01c4885f8cbe None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:10.011880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ebaa6-2ffc-4802-a732-01c4885f8cbe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:10.013326 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ebaa6-2ffc-4802-a732-01c4885f8cbe None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:10.030637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 225/441] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:10.033009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dac9e2e-9356-4cc6-b573-6b8952f1b293 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.033446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dac9e2e-9356-4cc6-b573-6b8952f1b293 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.033957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dac9e2e-9356-4cc6-b573-6b8952f1b293 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.036637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 226/442] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:10.037995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dac9e2e-9356-4cc6-b573-6b8952f1b293 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:10.053486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.054124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.054404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.054677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.055349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:10.055653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:10.057223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:10.071899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 217/443] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:10.074119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:10.176711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:10.200869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cacdd36-0232-4938-bee8-a53b7c633cd3 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:10.219063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dac9e2e-9356-4cc6-b573-6b8952f1b293 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:10.250586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd87c74-4275-4eca-b3a4-e63efa2cb5e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.251405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd87c74-4275-4eca-b3a4-e63efa2cb5e3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.251405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd87c74-4275-4eca-b3a4-e63efa2cb5e3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.251669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd87c74-4275-4eca-b3a4-e63efa2cb5e3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.262897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000e1222-c4e4-40bf-9d3d-de4643c83983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.263521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000e1222-c4e4-40bf-9d3d-de4643c83983 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.263804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000e1222-c4e4-40bf-9d3d-de4643c83983 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.264074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000e1222-c4e4-40bf-9d3d-de4643c83983 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.264776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000e1222-c4e4-40bf-9d3d-de4643c83983 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:10.265076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000e1222-c4e4-40bf-9d3d-de4643c83983 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:10.266508 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000e1222-c4e4-40bf-9d3d-de4643c83983 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:10.268801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd87c74-4275-4eca-b3a4-e63efa2cb5e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-bcd87c74-4275-4eca-b3a4-e63efa2cb5e3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:10.278603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:10.279745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:10.281280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 218/444] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:10.282408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd87c74-4275-4eca-b3a4-e63efa2cb5e3 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:10.282507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 227/445] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:10.286727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa8c34d-2e9e-4a1c-bccf-b21e6d96bbd3 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.286727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa8c34d-2e9e-4a1c-bccf-b21e6d96bbd3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.286727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa8c34d-2e9e-4a1c-bccf-b21e6d96bbd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.308923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:10.309800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-3fa8c34d-2e9e-4a1c-bccf-b21e6d96bbd3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:10.310905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3fa8c34d-2e9e-4a1c-bccf-b21e6d96bbd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:10.394233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 228/446] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:10.395124 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa8c34d-2e9e-4a1c-bccf-b21e6d96bbd3 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:10.427460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd87c74-4275-4eca-b3a4-e63efa2cb5e3 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:10.468674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.469469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.469747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.470040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.477721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:10.479976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:10.493816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:10.495104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:10.496121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:10.496121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:10.496121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:10.496121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:10.496121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:10.496121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:10.496121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:10.496121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:10.497161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 219/447] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:10.497161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:10.521768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:10.568482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa8c34d-2e9e-4a1c-bccf-b21e6d96bbd3 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:10.602951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:10.616233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.617907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.618601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.619149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] PATH_INFO: `/v3/users/glance` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.632034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:10.634600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:10.639349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae8c300-79f6-4f8d-8511-e1ff12b86297 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:10.659011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:10.660700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:10.661833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 229/448] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/users/glance => generated 84 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:10.662892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:10.689150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.690247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.690634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.691010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] PATH_INFO: `/v3/users/swift` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.699520 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:10.701183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:10.715123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:10.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 24 08:38:10.719307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:10.721468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 220/449] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/users/swift => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:10.722402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:10.842146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1b471d-1a56-4bd6-bd07-cd6f7294c1e5 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:10.852425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef06f7f-96fa-4927-bece-ff058acb2a01 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:10.881890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:10.888009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:10.894080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.894691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.894971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.895242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.895935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:10.896278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:10.896717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:10.897732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:10.897869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:10.898214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:10.898670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:10.899845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] RBAC: Adding query filter params (name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:10.900862 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:10.902845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:10.912394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 230/450] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:10.913268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:10.919644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 221/451] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:10.919644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:11.050652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d46b96-ca81-4732-9fda-646a2022b4af None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:11.063178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae04c9-6b8e-47b5-bcac-dadc20dcf8e7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:11.098477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:11.099185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:11.099587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:11.099983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:11.101145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:11.101893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:11.103904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:11.106672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:11.107399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:11.107674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:11.107941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:11.108819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:11.109150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:11.110742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:11.125301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 231/452] 10.222.0.39 () {56 vars in 1137 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:11.126158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:11.133931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 222/453] 10.222.0.39 () {56 vars in 1135 bytes} [Mon Oct 24 08:38:10 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:11.133931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:11.191348 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:11.203441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:11.256088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b37715-e2e5-4a58-9813-0d1633125180 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:11.271508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c45bf1-a976-4a07-94fb-bb94d0415d7b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:11.298276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dfad6b-3415-4b46-87b9-a957d604c955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:11.298921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dfad6b-3415-4b46-87b9-a957d604c955 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:11.299203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dfad6b-3415-4b46-87b9-a957d604c955 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:11.299476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dfad6b-3415-4b46-87b9-a957d604c955 None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:11.312921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dfad6b-3415-4b46-87b9-a957d604c955 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:11.314375 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:11.315031 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:11.315310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:11.315583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-58dfad6b-3415-4b46-87b9-a957d604c955 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:11.318121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:11.319632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 232/454] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:11.322183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dfad6b-3415-4b46-87b9-a957d604c955 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:11.333948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:11.335455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:11.352494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:11.355560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:11.358004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:11.358004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:11.358004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:11.358004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 223/455] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:11.358004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:11.477776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dfad6b-3415-4b46-87b9-a957d604c955 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:11.488288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:11.492267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51890d1a-bc18-48e6-9a7b-fd23c5c25078 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:11.495353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:11.524814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:11.525481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:11.525773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:11.526262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:11.527027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] RBAC: Adding query filter params (role.id=a1265c18dedb42d5943905f144e48fb7, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=4244858dfb464d029b6f6bed0491557d) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:11.527333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:11.529086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:11.532879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2c1f1-9eda-4805-a3c6-74b58cd86ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:11.533600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2c1f1-9eda-4805-a3c6-74b58cd86ec0 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:11.534001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2c1f1-9eda-4805-a3c6-74b58cd86ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:11.534280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2c1f1-9eda-4805-a3c6-74b58cd86ec0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:11.537560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 233/456] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:11 2022] GET /identity/v3/role_assignments?user.id=4244858dfb464d029b6f6bed0491557d&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=a1265c18dedb42d5943905f144e48fb7 => generated 608 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:11.538537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:11.629021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f2c1f1-9eda-4805-a3c6-74b58cd86ec0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:11.631959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f2c1f1-9eda-4805-a3c6-74b58cd86ec0 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:11.715059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6a722-2724-44b7-8df4-54e7252a050d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:11.778161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:11.779305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:11.779305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:11.779574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:11.794358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:11.796871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:11.817440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 224/457] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:38:11.819077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2c1f1-9eda-4805-a3c6-74b58cd86ec0 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:11.819289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:11.820783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:11.822290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:11.822290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:11.822290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:11.822290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 234/458] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:11.822290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:11.828637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:11.867135 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:11.977453 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84f38f-de35-4d1c-86a7-6f44eb1c1b83 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:11.987424 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2c1f1-9eda-4805-a3c6-74b58cd86ec0 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:12.027914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.028876 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.029216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.029557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.030588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:12.030935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.032754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:12.036864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.037478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.037765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.038017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.038681 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] RBAC: Adding query filter params (name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:12.038980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.040405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:12.054857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 235/459] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:12.054857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:12.055106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 225/460] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:12.055106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:12.149764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:12.202130 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:12.207382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82b197-c611-4c73-945e-6b253f1d7016 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:12.227401 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83f7c5-02ed-414f-891a-7c0e79163be6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:12.265003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.265907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.266496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.266957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.271015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.271628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.271902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.272258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.272849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:12.273145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.274544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:12.289283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 236/461] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:12.290127 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:12.290351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.294007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:12.319868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:12.321697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:12.324026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:12.324026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:12.324026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:12.324026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 226/462] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:12.324756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:12.491966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1556a2ad-0140-49af-9317-08970e6d471d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:12.499352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5df3847-de0d-42f4-9c6b-efa162889f9d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:12.522995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:12.546511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.547233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.547477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.547766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.563509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.566005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:12.570624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.571713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.572166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.572829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] PATH_INFO: `/v3/users/neutron` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.586070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:12.586872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:12.589366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:12.591695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:12.591695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:12.591695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:12.592017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:12.593497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 227/463] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:12.594580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:12.611034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:12.612686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 24 08:38:12.614264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:12.617499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 237/464] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:12 2022] GET /identity/v3/users/neutron => generated 85 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:12.619883 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:12.623966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:12.725064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c768f19-f751-43e3-b53b-440fb29b3371 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:12.774047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.774777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.775351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.775873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.787522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.789151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:12.792196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b725e7d-2fde-4ae7-b3f7-e2afb7fb992b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:12.806348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:12.808701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 228/465] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:12.810757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:12.850376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:12.860489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.861495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.861972 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.862429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.863603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:12.864231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.866707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:12.898638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 238/466] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:12.901053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:12.944617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8736ceb-c444-429e-b20d-e2318b555419 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:12.985241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:12.986357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:12.986357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:12.986672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:12.990400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:12.990400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:12.990400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:13.002623 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:13.014412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 229/467] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:13.015321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:13.073654 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdfb08c-698b-4571-936c-584aaa5a6a8c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:13.131658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:13.132322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:13.132596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:13.133063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:13.133721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:13.135035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:13.135452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:13.137650 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:13.152412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cfb062-b460-4c09-a3b4-18203b0a8e16 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:13.170095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 239/468] 10.222.0.39 () {56 vars in 1139 bytes} [Mon Oct 24 08:38:12 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:13.170095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:13.203701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:13.204418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:13.204717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:13.205002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:13.206279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:13.206766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:13.209105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:13.231035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 230/469] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:13.232251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:13.309939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faad7ad-ab4c-449c-87f7-9e9d5ba377a4 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:13.341029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:13.360752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:13.361382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:13.361657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:13.361940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:13.370989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:13.373129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:13.379161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126540ee-68cc-470b-af80-13080c50b5dc None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:13.395564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:13.397295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:13.398577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:13.398577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:13.398577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:13.398577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:13.398577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:13.398577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:13.398577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:13.398577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:13.403547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 240/470] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:13.404902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:13.429807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:13.430755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:13.431191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:13.431592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:13.442455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:13.444523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:13.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:13.460846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 231/471] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:13 2022] GET /identity/v3/users/cinder => generated 84 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:13.462051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:13.475453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:13.585385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ee4cf9-faf5-410e-bcb4-43796e583f38 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:13.621098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9878e5-d935-4cc9-975d-2d075aae7712 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:13.633001 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1726df37-b97b-4ce4-b3a1-d06789e14e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:13.633893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1726df37-b97b-4ce4-b3a1-d06789e14e5e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:13.634345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1726df37-b97b-4ce4-b3a1-d06789e14e5e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:13.634722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1726df37-b97b-4ce4-b3a1-d06789e14e5e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:13.685047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5f71b-d8c0-4bd0-98e3-fa82edf6b8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:13.686219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5f71b-d8c0-4bd0-98e3-fa82edf6b8a0 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:13.686695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5f71b-d8c0-4bd0-98e3-fa82edf6b8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:13.687139 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5f71b-d8c0-4bd0-98e3-fa82edf6b8a0 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/60a07d37339f4f83ad2ce8be3d0892a8/roles/a1265c18dedb42d5943905f144e48fb7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:13.706913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:13.708390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae5f71b-d8c0-4bd0-98e3-fa82edf6b8a0 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=60a07d37339f4f83ad2ce8be3d0892a8, role_id=a1265c18dedb42d5943905f144e48fb7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:13.709922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae5f71b-d8c0-4bd0-98e3-fa82edf6b8a0 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:13.732198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 232/472] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:13 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/60a07d37339f4f83ad2ce8be3d0892a8/roles/a1265c18dedb42d5943905f144e48fb7 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:13.732198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5f71b-d8c0-4bd0-98e3-fa82edf6b8a0 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:13.750886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1726df37-b97b-4ce4-b3a1-d06789e14e5e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:13.752945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1726df37-b97b-4ce4-b3a1-d06789e14e5e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:13.838779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:13.926422 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5f71b-d8c0-4bd0-98e3-fa82edf6b8a0 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:13.948184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 241/473] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 544 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:38:13.951545 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1726df37-b97b-4ce4-b3a1-d06789e14e5e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:14.010613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:14.012417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:14.013077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:14.013690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:14.014977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:14.015493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:14.018399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:14.043606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 233/474] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:14.043606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:14.092935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:14.121508 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1726df37-b97b-4ce4-b3a1-d06789e14e5e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:14.168177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49843c30-f1b7-4057-b654-5a48a3b4fb90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:14.168450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49843c30-f1b7-4057-b654-5a48a3b4fb90 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:14.168585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49843c30-f1b7-4057-b654-5a48a3b4fb90 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:14.168963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49843c30-f1b7-4057-b654-5a48a3b4fb90 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:14.170011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49843c30-f1b7-4057-b654-5a48a3b4fb90 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:14.170569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49843c30-f1b7-4057-b654-5a48a3b4fb90 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:14.172195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49843c30-f1b7-4057-b654-5a48a3b4fb90 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:14.181209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee21e31-8ed7-43bd-914b-07e4cdb1fbb9 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:14.205019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:14.210417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 242/475] 10.222.0.39 () {56 vars in 1137 bytes} [Mon Oct 24 08:38:13 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:14.213741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b7239a-41ad-4405-a63b-9002f69bc803 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:14.214498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b7239a-41ad-4405-a63b-9002f69bc803 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:14.214885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b7239a-41ad-4405-a63b-9002f69bc803 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:14.217393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 243/476] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:14.220222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b7239a-41ad-4405-a63b-9002f69bc803 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:14.233623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:14.234598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:14.234884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:14.235155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:14.248919 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:14.250547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:14.269605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:14.271328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:14.272922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:14.272922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:14.272922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:14.272922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 234/477] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:14.272922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:14.405674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b7239a-41ad-4405-a63b-9002f69bc803 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:14.442335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901b98ab-8575-44e5-8a82-d63171efe046 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:14.469313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1d4344-4b6a-4cbf-bf41-80c3271db1d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:14.470061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d4344-4b6a-4cbf-bf41-80c3271db1d7 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:14.470505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d4344-4b6a-4cbf-bf41-80c3271db1d7 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:14.470933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d4344-4b6a-4cbf-bf41-80c3271db1d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:14.472451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d4344-4b6a-4cbf-bf41-80c3271db1d7 None admin] RBAC: Adding query filter params (role.id=a1265c18dedb42d5943905f144e48fb7, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=5a4fd11d5d85479c9aa1d87b23a13141) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:14.472982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d4344-4b6a-4cbf-bf41-80c3271db1d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:14.476754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d4344-4b6a-4cbf-bf41-80c3271db1d7 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:14.486919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 244/478] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:14 2022] GET /identity/v3/role_assignments?user.id=5a4fd11d5d85479c9aa1d87b23a13141&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=a1265c18dedb42d5943905f144e48fb7 => generated 608 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:14.488910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca609c-fecc-41de-a545-ef8ba0e5cb4e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:14.489506 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca609c-fecc-41de-a545-ef8ba0e5cb4e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:14.490053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca609c-fecc-41de-a545-ef8ba0e5cb4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:14.494992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43607708-6179-4ba6-a5b6-7fe76717aeef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:14.496167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43607708-6179-4ba6-a5b6-7fe76717aeef None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:14.496658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43607708-6179-4ba6-a5b6-7fe76717aeef None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:14.497198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43607708-6179-4ba6-a5b6-7fe76717aeef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:14.499630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:14.515890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-67ca609c-fecc-41de-a545-ef8ba0e5cb4e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:14.517699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-67ca609c-fecc-41de-a545-ef8ba0e5cb4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:14.569841 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:14.617128 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43607708-6179-4ba6-a5b6-7fe76717aeef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:14.618986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43607708-6179-4ba6-a5b6-7fe76717aeef None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:14.660751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 245/479] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:14.661805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca609c-fecc-41de-a545-ef8ba0e5cb4e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:14.803085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 235/480] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 531 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:38:14.817230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43607708-6179-4ba6-a5b6-7fe76717aeef None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:14.882681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca609c-fecc-41de-a545-ef8ba0e5cb4e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:14.927274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:14.938218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:14.938841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:14.939122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:14.939414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:14.940106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:14.940442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:14.942236 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:14.959455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 246/481] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:14.965678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:14.967741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:15.039302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43607708-6179-4ba6-a5b6-7fe76717aeef None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:15.107766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cc4840-6fbb-4198-9002-023a508f3d87 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:15.112435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:15.114064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:15.115079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:15.115909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:15.132889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:15.135816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:15.161247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:15.162281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:15.162769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:15.163247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:15.168049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:15.169149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:15.170203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:15.170203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:15.170203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:15.170819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 236/482] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:15.172220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:15.177111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:15.179516 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:15.198055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:15.199308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:15.200497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:15.200497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:15.200497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:15.200497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:15.200497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:15.200497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:15.200497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:15.200497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:15.203005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 247/483] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:15.209737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:15.300891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:15.363199 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c883b24-b09a-475b-94e5-03a723fc5bd3 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:15.375743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95b5f18-68e2-42c5-b6ed-bf1870345648 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:15.406939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:15.429665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:15.430827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:15.431162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1549be-7edf-4fee-a5cf-7f6f605dec4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:15.431615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:15.432696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:15.433224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1549be-7edf-4fee-a5cf-7f6f605dec4c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:15.433766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:15.434133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:15.435586 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:15.436264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1549be-7edf-4fee-a5cf-7f6f605dec4c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:15.436869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1549be-7edf-4fee-a5cf-7f6f605dec4c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:15.453215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 248/484] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:15.454304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:15.459580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1549be-7edf-4fee-a5cf-7f6f605dec4c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-3d1549be-7edf-4fee-a5cf-7f6f605dec4c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:15.467971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:15.468991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:38:15.469803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:15.475191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 237/485] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:38:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:15.476948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1549be-7edf-4fee-a5cf-7f6f605dec4c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:15.617452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9948dd-f99c-4e3b-abaa-c05af034930a None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:15.635561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:15.663725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:15.664096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1549be-7edf-4fee-a5cf-7f6f605dec4c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:15.664878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:15.665319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:15.665738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:15.667335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:15.667836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:15.669570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:15.685008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 249/486] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:15.686169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:15.727621 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:15.728347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:15.729855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:15.730178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:15.742865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:15.745634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:15.763697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:15.765434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 238/487] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:15.832017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:15.869413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbef9fb-4d08-40e7-a397-03e05fadb547 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:15.912294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4551567a-9fb3-4259-a5b3-84f0e812dc3a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:15.944103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:15.945662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:15.946370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:15.947030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:15.949088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:15.949953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:15.953705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:15.961656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:15.962336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:15.962692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:15.963033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:15.963879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] RBAC: Adding query filter params (role.id=a1265c18dedb42d5943905f144e48fb7, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=8bfae94e0fa041c9802729370a810245) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:15.964346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:15.966192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:15.973567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 250/488] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:15.976216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:15.986241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 239/489] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:15 2022] GET /identity/v3/role_assignments?user.id=8bfae94e0fa041c9802729370a810245&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=a1265c18dedb42d5943905f144e48fb7 => generated 608 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:15.987827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:16.063149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:16.311341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a561fb2-594f-42dd-a4b1-907395f68aa7 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:16.389680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09df372d-c327-4cdc-bd2f-10f71b10f2b2 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:16.401594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:16.419322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:16.421702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:16.422421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:16.423109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:16.424500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:16.425690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:16.429350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:16.450634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:16.451993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:16.452543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:16.453045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:16.457611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 251/490] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 482 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:16.468730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:16.469277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:16.472961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:16.497963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:16.499734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:16.502815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:16.502815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:16.502815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:16.506562 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 240/491] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 520 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:16.515773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:16.724013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:16.747698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b110dd0f-8b9d-43d1-a2c0-f2f78cab45df None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:16.818836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf835b4-42cf-4de4-85d7-1cf19314be3f None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:16.842666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:16.844028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:16.844603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:16.845167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:16.846389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] RBAC: Adding query filter params (role.id=a1265c18dedb42d5943905f144e48fb7, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=412f520f90564e44819e8bf947992efb) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:16.849550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:16.851702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:16.870215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 252/492] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:16 2022] GET /identity/v3/role_assignments?user.id=412f520f90564e44819e8bf947992efb&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=a1265c18dedb42d5943905f144e48fb7 => generated 608 bytes in 402 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:16.896988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbda614-8264-47d4-b67e-1f90ce1efdb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:16.898006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbda614-8264-47d4-b67e-1f90ce1efdb2 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:16.898559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbda614-8264-47d4-b67e-1f90ce1efdb2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:16.899189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbda614-8264-47d4-b67e-1f90ce1efdb2 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:16.900307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbda614-8264-47d4-b67e-1f90ce1efdb2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:16.901941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbda614-8264-47d4-b67e-1f90ce1efdb2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:16.904253 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbda614-8264-47d4-b67e-1f90ce1efdb2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:16.923112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 241/493] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 408 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:16.930410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:17.069298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244792d-0ac2-442d-bf9f-97f33e1fb66f None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:17.069683 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244792d-0ac2-442d-bf9f-97f33e1fb66f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:17.069881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244792d-0ac2-442d-bf9f-97f33e1fb66f None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:17.071543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 242/494] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:17.078829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa837e34-7e64-452f-aee3-2f6d4473c166 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:17.079154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa837e34-7e64-452f-aee3-2f6d4473c166 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:17.079360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa837e34-7e64-452f-aee3-2f6d4473c166 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:17.130038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee08100-4905-4fe7-8a5e-7b3abd04986c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:17.133246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-fa837e34-7e64-452f-aee3-2f6d4473c166 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:17.134663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-fa837e34-7e64-452f-aee3-2f6d4473c166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:17.153365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:17.190575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:17.191347 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:17.191887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:17.192271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] PATH_INFO: `/v3/users/nova` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:17.206496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:17.207921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:17.223704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:17.224890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:17.230857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 253/495] 10.222.0.39 () {56 vars in 1093 bytes} [Mon Oct 24 08:38:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:17.233414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:17.268305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:17.342808 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 243/496] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:38:17.411902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bec2e6-df57-4386-81ee-7fd29c48d42d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:17.464840 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c4c44-1e15-4d67-8f38-e5854a069809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:17.465470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c4c44-1e15-4d67-8f38-e5854a069809 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:17.465751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c4c44-1e15-4d67-8f38-e5854a069809 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:17.466024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c4c44-1e15-4d67-8f38-e5854a069809 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:17.466771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348c4c44-1e15-4d67-8f38-e5854a069809 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:17.467075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348c4c44-1e15-4d67-8f38-e5854a069809 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:17.468554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348c4c44-1e15-4d67-8f38-e5854a069809 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:17.495849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 254/497] 10.222.0.39 () {56 vars in 1133 bytes} [Mon Oct 24 08:38:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:17.498087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa837e34-7e64-452f-aee3-2f6d4473c166 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:17.513843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb85d69-b89a-4833-a425-62f7d4051091 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:17.514154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb85d69-b89a-4833-a425-62f7d4051091 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:17.514389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb85d69-b89a-4833-a425-62f7d4051091 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:17.515887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 255/498] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:17.524778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35793353-2f94-4f35-a9e2-6a5fe577df24 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:17.524778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35793353-2f94-4f35-a9e2-6a5fe577df24 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:17.524778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35793353-2f94-4f35-a9e2-6a5fe577df24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:17.548765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-35793353-2f94-4f35-a9e2-6a5fe577df24 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:17.551853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-35793353-2f94-4f35-a9e2-6a5fe577df24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:17.692692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa837e34-7e64-452f-aee3-2f6d4473c166 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:17.705455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 256/499] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:17.750058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35793353-2f94-4f35-a9e2-6a5fe577df24 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:17.759784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:17.769936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:17.770166 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:17.770576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:17.770902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:17.801656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:17.803320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:17.822323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:17.842079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:17.844301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:17.844301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:17.844301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:17.844301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 244/500] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:17.844301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:18.003930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35793353-2f94-4f35-a9e2-6a5fe577df24 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:18.085418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b41586b-2e8e-4455-9dd8-71d57a2b1c8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:18.086072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b41586b-2e8e-4455-9dd8-71d57a2b1c8e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:18.086356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b41586b-2e8e-4455-9dd8-71d57a2b1c8e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:18.086632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b41586b-2e8e-4455-9dd8-71d57a2b1c8e None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:18.109136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b41586b-2e8e-4455-9dd8-71d57a2b1c8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-6b41586b-2e8e-4455-9dd8-71d57a2b1c8e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:18.120697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:18.121795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:18.123071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:18.124264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 257/501] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 374 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:18.125454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b41586b-2e8e-4455-9dd8-71d57a2b1c8e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:18.132590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa1b651-b625-43b0-89a3-fc574e59f825 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:18.209581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:18.210688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:18.211438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:18.212033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:18.214625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:18.215034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:18.217275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:18.232806 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:18.241891 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 245/502] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 399 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:18.243633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:18.283016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b41586b-2e8e-4455-9dd8-71d57a2b1c8e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:18.335012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:18.336313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:18.336612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:18.337363 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:18.340201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:18.357763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:18.360518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:18.385013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:18.387269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:18.389712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:18.389712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:18.389712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:18.389712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 258/503] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:18.389712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:18.401685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cda83-0dad-4071-b50a-7db979a72299 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:18.457181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:18.458198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:18.458669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:18.459243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:18.460327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] RBAC: Adding query filter params (name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:18.460765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:18.462666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:18.476763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 246/504] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:18.478514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:18.547620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a343b7b9-a861-4291-b08e-21863bc1a961 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:18.592943 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:18.598300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:18.599147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:18.599507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:18.599890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:18.611463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:18.614117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:18.636576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:18.639035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:18.642351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:18.642351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:18.642351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:18.642351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:18.642351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:18.642351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:18.642351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:18.642351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:18.646840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42657c3-8467-41f0-bb93-a8ffb8fc0daa None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:18.647573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 259/505] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:18.652206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:18.677364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:18.694622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:18.695540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:18.695837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:18.696207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:18.697905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:18.699340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:18.701306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:18.724488 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 247/506] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:18.725927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:18.834848 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf273ca6-f1b8-4a44-b19f-d8e29dbe5222 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:18.888884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:18.889948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:18.890236 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:18.890658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:18.907906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d43390-0cb4-4597-a6c4-fb8f57c5076a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:18.910301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:18.912591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:18.937906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:18.939821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:18.941611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:18.941611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:18.941611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:18.944381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 260/507] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:18.945850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:18.951254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:18.951877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:18.952224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:18.952467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:18.954092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:18.954407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:18.955869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:18.975795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 248/508] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:18.975795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:19.013248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:19.027781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:19.122546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9a292f-78d2-4ac9-a1c0-b48f3e997e4e None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:19.129451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bd8c2c-af1f-4166-83cf-511b47b79dde None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:19.179646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:19.180340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:19.180629 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:19.180906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:19.182395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:19.182705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:19.184761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:19.188741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e407e2d-e630-4c65-b6d0-0547bc089636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:19.190270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e407e2d-e630-4c65-b6d0-0547bc089636 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:19.191076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e407e2d-e630-4c65-b6d0-0547bc089636 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:19.191817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e407e2d-e630-4c65-b6d0-0547bc089636 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:19.193387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e407e2d-e630-4c65-b6d0-0547bc089636 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:19.196638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e407e2d-e630-4c65-b6d0-0547bc089636 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:19.202140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 249/509] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:19.203288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:19.236309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 261/510] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:38:18 2022] POST /identity/v3/users => generated 254 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:19.238574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e407e2d-e630-4c65-b6d0-0547bc089636 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:19.435465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985fc7b2-b611-403b-9e0e-791b1e68726b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:19.438356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:19.476664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:19.514830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e407e2d-e630-4c65-b6d0-0547bc089636 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:19.531696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dcc87e-e466-40fe-bdab-3ed9018644ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:19.532865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcc87e-e466-40fe-bdab-3ed9018644ca None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:19.533152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcc87e-e466-40fe-bdab-3ed9018644ca None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:19.533425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcc87e-e466-40fe-bdab-3ed9018644ca None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/4244858dfb464d029b6f6bed0491557d/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:19.535903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dcc87e-e466-40fe-bdab-3ed9018644ca None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=4244858dfb464d029b6f6bed0491557d, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:19.537559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dcc87e-e466-40fe-bdab-3ed9018644ca None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:19.563632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 250/511] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:19 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/4244858dfb464d029b6f6bed0491557d/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:19.577601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6696490-5a32-4f13-af3b-f37e05f71cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:19.578310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6696490-5a32-4f13-af3b-f37e05f71cdf None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:19.578593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6696490-5a32-4f13-af3b-f37e05f71cdf None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:19.578929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6696490-5a32-4f13-af3b-f37e05f71cdf None admin] PATH_INFO: `/v3/users/swift` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:19.590815 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6696490-5a32-4f13-af3b-f37e05f71cdf None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:19.592440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6696490-5a32-4f13-af3b-f37e05f71cdf None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-b6696490-5a32-4f13-af3b-f37e05f71cdf None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:19.610032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:19.611138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 24 08:38:19.612355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:19.620627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 262/512] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:38:19 2022] GET /identity/v3/users/swift => generated 83 bytes in 375 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:19.623465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dcc87e-e466-40fe-bdab-3ed9018644ca None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:19.822575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dcc87e-e466-40fe-bdab-3ed9018644ca None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:19.872961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642acd09-6c8f-4110-99cb-13b3cb63d862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:19.874282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642acd09-6c8f-4110-99cb-13b3cb63d862 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:19.874901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642acd09-6c8f-4110-99cb-13b3cb63d862 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:19.875605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642acd09-6c8f-4110-99cb-13b3cb63d862 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:19.876835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642acd09-6c8f-4110-99cb-13b3cb63d862 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:19.877590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642acd09-6c8f-4110-99cb-13b3cb63d862 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:19.879614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642acd09-6c8f-4110-99cb-13b3cb63d862 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:19.908491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae8fd9-0917-411e-8bbf-f0ca6b5145bf None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:19.908748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae8fd9-0917-411e-8bbf-f0ca6b5145bf None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:19.909007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae8fd9-0917-411e-8bbf-f0ca6b5145bf None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:19.914847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 263/513] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:19.916932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c6fad2-c01f-4a12-96c6-412114e9009a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:19.917288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c6fad2-c01f-4a12-96c6-412114e9009a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:19.917563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c6fad2-c01f-4a12-96c6-412114e9009a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:19.918997 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 251/514] 10.222.0.39 () {56 vars in 1135 bytes} [Mon Oct 24 08:38:19 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:19.920892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d016-cc64-40b9-b887-9be0cacba8b1 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:19.921443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d016-cc64-40b9-b887-9be0cacba8b1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:19.921918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d016-cc64-40b9-b887-9be0cacba8b1 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:19.923949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 252/515] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:19.925276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be16d016-cc64-40b9-b887-9be0cacba8b1 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:19.939962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-06c6fad2-c01f-4a12-96c6-412114e9009a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:19.941209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-06c6fad2-c01f-4a12-96c6-412114e9009a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:19.953361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:20.084757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 264/516] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:38:20.086602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a711f8d6-c5df-4276-b8ce-433278e2dfa8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:20.087043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a711f8d6-c5df-4276-b8ce-433278e2dfa8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:20.087564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a711f8d6-c5df-4276-b8ce-433278e2dfa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:20.116646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be16d016-cc64-40b9-b887-9be0cacba8b1 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:20.141531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a711f8d6-c5df-4276-b8ce-433278e2dfa8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:20.142794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a711f8d6-c5df-4276-b8ce-433278e2dfa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:20.187812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1bbc0a-f94a-4635-8521-0fc3253ad4cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:20.188885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1bbc0a-f94a-4635-8521-0fc3253ad4cc None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:20.189433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1bbc0a-f94a-4635-8521-0fc3253ad4cc None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:20.190050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1bbc0a-f94a-4635-8521-0fc3253ad4cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:20.307810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1bbc0a-f94a-4635-8521-0fc3253ad4cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:20.309367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1bbc0a-f94a-4635-8521-0fc3253ad4cc None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:20.334016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:20.367120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 265/517] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:20.459308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 253/518] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 24 08:38:20.466637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a711f8d6-c5df-4276-b8ce-433278e2dfa8 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:20.478153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1bbc0a-f94a-4635-8521-0fc3253ad4cc None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:20.490066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:20.680883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1bbc0a-f94a-4635-8521-0fc3253ad4cc None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:20.725343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:20.744665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a711f8d6-c5df-4276-b8ce-433278e2dfa8 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:20.758895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:20.759792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:20.760454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:20.760625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] PATH_INFO: `/v3/services/network` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:20.761459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:20.763160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:20.782890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:20.784565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:20.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 254/519] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:20 2022] GET /identity/v3/services/network => generated 88 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:20.793438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:20.826378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:20.827248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:20.827540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:20.827942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:20.841770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:20.843318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:20.862745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:20.865873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:20.869673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:20.869673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:20.869673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:20.869673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 266/520] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 399 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:20.869673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:20.941675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:20.995579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece752a-a8b2-4b86-ab80-6efe6fc1e167 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:21.025009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e74c8-56d3-476e-9fe5-e7010623e5e7 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:21.035251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:21.036315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:21.036602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:21.036872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:21.037505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] RBAC: Adding query filter params (name=network) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:21.037810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:21.039193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:21.050776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 255/521] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:20 2022] GET /identity/v3/services?name=network => generated 124 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:21.051726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:21.077701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:21.079141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:21.079697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:21.080322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:21.081360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:21.083494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:21.098841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:21.100583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:21.101798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:21.106058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 267/522] 10.222.0.39 () {56 vars in 1129 bytes} [Mon Oct 24 08:38:20 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:21.107004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:21.128857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:21.205646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb1f2b-2dc1-4d33-ba99-dd5c57785ea5 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:21.255766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44853f08-4665-4303-9fa8-d72928df7411 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:21.266832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:21.267736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:21.268871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:21.269335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:21.270148 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:21.270594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:21.272581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:21.301299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 256/523] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:21.301534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:21.309460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:21.329911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:21.330918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:21.331420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:21.331841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:21.332549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] RBAC: Adding query filter params (type=network) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:21.332908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:21.334696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:21.348400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 268/524] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:21 2022] GET /identity/v3/services?type=network => generated 124 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:21.349997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:21.519777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb78f21-00e5-46ed-a98c-6bfc2b0c7b1d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:21.542000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:21.566013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d3ddcd-36f0-41a8-839d-7762444684af None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:21.614287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:21.614948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:21.615257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:21.615530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:21.616220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:21.616552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:21.618171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:21.625604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 257/525] 10.222.0.39 () {56 vars in 1124 bytes} [Mon Oct 24 08:38:21 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:21.631386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:21.633356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:21.634096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:21.634689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:21.635937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:21.650285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:21.653406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:21.678087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:21.679574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:21.681004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:21.681004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:21.681004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:21.681004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:21.681004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:21.681004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:21.681004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:21.681004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:21.685748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 269/526] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:21.693425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:21.779885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:21.853139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee8c581-89e1-48c5-af8b-51683598d1f8 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:21.875142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caad760-f913-4f4f-ae5b-61fb38a8a9b6 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:21.896606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:21.897666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:21.898148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:21.898639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:21.899659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:21.900196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:21.902728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:21.917796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 270/527] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:21.924036 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:21.965615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d98f6-0888-4958-bab3-31ce16dc80cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:21.966666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d98f6-0888-4958-bab3-31ce16dc80cc None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:21.967154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d98f6-0888-4958-bab3-31ce16dc80cc None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:21.967627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d98f6-0888-4958-bab3-31ce16dc80cc None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:21.968614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d98f6-0888-4958-bab3-31ce16dc80cc None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:21.969138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d98f6-0888-4958-bab3-31ce16dc80cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:21.970757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d98f6-0888-4958-bab3-31ce16dc80cc None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:22.002561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 258/528] 10.222.0.39 () {56 vars in 1124 bytes} [Mon Oct 24 08:38:21 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 369 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:22.050362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d37df-066b-49b6-be57-afbaf00d5710 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:22.059871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:22.099097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58da5755-2f48-4c6a-b81f-ab286d4eebdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:22.100116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58da5755-2f48-4c6a-b81f-ab286d4eebdd None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:22.100700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58da5755-2f48-4c6a-b81f-ab286d4eebdd None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:22.101143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58da5755-2f48-4c6a-b81f-ab286d4eebdd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:22.102081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58da5755-2f48-4c6a-b81f-ab286d4eebdd None admin] RBAC: Adding query filter params (role.id=a1265c18dedb42d5943905f144e48fb7, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=60a07d37339f4f83ad2ce8be3d0892a8) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:22.102592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58da5755-2f48-4c6a-b81f-ab286d4eebdd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:22.104269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58da5755-2f48-4c6a-b81f-ab286d4eebdd None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:22.113558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 271/529] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:21 2022] GET /identity/v3/role_assignments?user.id=60a07d37339f4f83ad2ce8be3d0892a8&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=a1265c18dedb42d5943905f144e48fb7 => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:22.344362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608873a4-3fc8-4b1c-a5f6-f8bf9b5126a4 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:22.344779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608873a4-3fc8-4b1c-a5f6-f8bf9b5126a4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:22.345167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608873a4-3fc8-4b1c-a5f6-f8bf9b5126a4 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:22.347051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 259/530] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:22.353032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be53e313-39a8-473d-9f5b-7134d146c7a4 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:22.353480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be53e313-39a8-473d-9f5b-7134d146c7a4 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:22.354945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be53e313-39a8-473d-9f5b-7134d146c7a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:22.380248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-be53e313-39a8-473d-9f5b-7134d146c7a4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:22.381555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-be53e313-39a8-473d-9f5b-7134d146c7a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:22.396516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34637a2-37ec-435b-b678-76803f6d180c None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:22.396881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34637a2-37ec-435b-b678-76803f6d180c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:22.397099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34637a2-37ec-435b-b678-76803f6d180c None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:22.398856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 260/531] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:22.406366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3ed8bd-a139-4a97-b8d8-fca2b6366fd1 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:22.406664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3ed8bd-a139-4a97-b8d8-fca2b6366fd1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:22.407066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3ed8bd-a139-4a97-b8d8-fca2b6366fd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:22.431336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7d3ed8bd-a139-4a97-b8d8-fca2b6366fd1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:22.435296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7d3ed8bd-a139-4a97-b8d8-fca2b6366fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:22.481412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 272/532] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:22.484032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:22.538053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 261/533] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 24 08:38:22.813477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be53e313-39a8-473d-9f5b-7134d146c7a4 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:22.902058 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3ed8bd-a139-4a97-b8d8-fca2b6366fd1 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:22.927405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:22.931556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be53e313-39a8-473d-9f5b-7134d146c7a4 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:22.977426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d1fc4f-5398-4c4d-af9f-e92bccdf5daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:22.978072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d1fc4f-5398-4c4d-af9f-e92bccdf5daf None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:22.978378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d1fc4f-5398-4c4d-af9f-e92bccdf5daf None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:22.978715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d1fc4f-5398-4c4d-af9f-e92bccdf5daf None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:22.997165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d1fc4f-5398-4c4d-af9f-e92bccdf5daf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-02d1fc4f-5398-4c4d-af9f-e92bccdf5daf None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:23.002864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:23.004999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:23.006982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:23.013497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 273/534] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:23.013497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d1fc4f-5398-4c4d-af9f-e92bccdf5daf None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:23.112834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3ed8bd-a139-4a97-b8d8-fca2b6366fd1 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:23.155869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9218460-e960-427e-88d5-3fd41a6049f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:23.156527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9218460-e960-427e-88d5-3fd41a6049f5 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:23.156782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9218460-e960-427e-88d5-3fd41a6049f5 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:23.157051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9218460-e960-427e-88d5-3fd41a6049f5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:23.179982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9218460-e960-427e-88d5-3fd41a6049f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-d9218460-e960-427e-88d5-3fd41a6049f5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:23.185090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:23.186213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:23.187260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 262/535] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:38:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:23.196474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9218460-e960-427e-88d5-3fd41a6049f5 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:23.211571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d1fc4f-5398-4c4d-af9f-e92bccdf5daf None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:23.236771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:23.251798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:23.253285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:23.253995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:23.254705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:23.256092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] RBAC: Adding query filter params (name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:23.256651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:23.258438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:23.273882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 274/536] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:23.285651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:23.321287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9218460-e960-427e-88d5-3fd41a6049f5 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:23.331854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:23.373879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:23.374507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:23.374781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:23.375049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:23.375716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:23.376015 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:23.377551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:23.393886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 263/537] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:23.403965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:23.433676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d692a6-076d-429c-a167-7c088c0c1942 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:23.509308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:23.510012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:23.510276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:23.510549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:23.530346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:23.533435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:23.577024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:23.580159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:23.582945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:23.582945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:23.582945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:23.588399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 275/538] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:23.588399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:23.598741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:23.654924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53a1e3c-79cf-4071-b50a-4bca2e9c42a4 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:23.688923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:23.711559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d465639-7261-4e94-a81b-add872333d5a None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:23.753675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:23.754331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:23.754606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:23.754882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:23.755549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:23.755865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:23.757266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:23.759584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:23.760398 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:23.760668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:23.760933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:23.784855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:23.786386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:23.795527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 276/539] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:23.797512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:23.823599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:23.825525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:23.826814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:23.826814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:23.826814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:23.832158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 264/540] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 422 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:23.833806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:23.982660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3e0f20-5c67-4324-8065-a1f2db96c476 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:24.047239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:24.047868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:24.048177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:24.048438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:24.052236 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:24.061697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:24.063151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:24.069779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336bb750-d477-4b8f-a725-b2eb8e0e824b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:24.093147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:24.094936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:24.097412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 277/541] 10.222.0.39 () {56 vars in 1117 bytes} [Mon Oct 24 08:38:23 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:24.108850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:24.138973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:24.139816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:24.139816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:24.140098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:24.140760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:24.141063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:24.142438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:24.159393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:24.166734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 265/542] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:24.177888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:24.314838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edd1476-4a80-41e8-9dc5-1bf85aede0c5 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:24.346080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9120b319-2c94-405d-af22-e305e26626a3 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:24.378158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:24.378485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:24.378641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:24.378988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:24.379787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:24.382264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:24.382264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:24.421136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 278/543] 10.222.0.39 () {56 vars in 1149 bytes} [Mon Oct 24 08:38:24 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:24.429734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:24.430380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:24.430659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:24.430976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] PATH_INFO: `/v3/users/nova` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:24.432097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:24.446601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:24.448506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:24.471257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:24.481029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:24.482873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 266/544] 10.222.0.39 () {56 vars in 1093 bytes} [Mon Oct 24 08:38:24 2022] GET /identity/v3/users/nova => generated 82 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:24.501251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:24.554095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:24.629654 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:24.642836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0009a1-8507-4926-aff3-0e9c39389a68 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:24.715986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:24.717313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:24.718471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:24.719115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:24.741229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:24.745009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:24.758366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743d57e7-9af4-44a2-ad89-eeb15cdcdf32 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:24.778308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:24.779701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:24.780909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:24.780909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:24.780909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:24.786111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 279/545] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:24.791113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:24.865522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc05e341-b82c-45a4-8597-e7469fa99fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:24.866323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc05e341-b82c-45a4-8597-e7469fa99fb0 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:24.866434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc05e341-b82c-45a4-8597-e7469fa99fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:24.866675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc05e341-b82c-45a4-8597-e7469fa99fb0 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:24.867489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc05e341-b82c-45a4-8597-e7469fa99fb0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:24.867745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc05e341-b82c-45a4-8597-e7469fa99fb0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:24.869712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc05e341-b82c-45a4-8597-e7469fa99fb0 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:24.904410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 267/546] 10.222.0.39 () {56 vars in 1133 bytes} [Mon Oct 24 08:38:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 397 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:24.907126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c58bf9f-4495-4bce-8a82-acdd3477d014 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:24.907585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c58bf9f-4495-4bce-8a82-acdd3477d014 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:24.908077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c58bf9f-4495-4bce-8a82-acdd3477d014 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:24.912662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 268/547] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:24.915185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c58bf9f-4495-4bce-8a82-acdd3477d014 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:24.952104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb39f5c-541d-42ae-8b36-fe4aa66bd59f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:24.955863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:25.004202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6e275f-9cbd-4c2f-96bc-7b65c04052b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:25.005246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e275f-9cbd-4c2f-96bc-7b65c04052b4 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.005246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e275f-9cbd-4c2f-96bc-7b65c04052b4 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.005548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e275f-9cbd-4c2f-96bc-7b65c04052b4 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.006043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6e275f-9cbd-4c2f-96bc-7b65c04052b4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:25.006431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6e275f-9cbd-4c2f-96bc-7b65c04052b4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:25.007812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6e275f-9cbd-4c2f-96bc-7b65c04052b4 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:25.032166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 280/548] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:25.032166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94346d9e-9844-46dd-b33b-3160b096a23f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.032166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94346d9e-9844-46dd-b33b-3160b096a23f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.032166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94346d9e-9844-46dd-b33b-3160b096a23f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.051032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-94346d9e-9844-46dd-b33b-3160b096a23f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:25.052255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-94346d9e-9844-46dd-b33b-3160b096a23f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:25.126729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c58bf9f-4495-4bce-8a82-acdd3477d014 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:25.151668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:25.153722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 281/549] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:25.156166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94346d9e-9844-46dd-b33b-3160b096a23f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:25.173361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785e5b5c-d105-4dab-a089-498f20b7322b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:25.173981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785e5b5c-d105-4dab-a089-498f20b7322b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.174264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785e5b5c-d105-4dab-a089-498f20b7322b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.174541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785e5b5c-d105-4dab-a089-498f20b7322b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.282686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785e5b5c-d105-4dab-a089-498f20b7322b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:25.285357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785e5b5c-d105-4dab-a089-498f20b7322b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:25.313447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94346d9e-9844-46dd-b33b-3160b096a23f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:25.316818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:25.374574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdd852f-1dc1-46a1-a703-beb7bac3f4d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:25.375806 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd852f-1dc1-46a1-a703-beb7bac3f4d6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.380197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd852f-1dc1-46a1-a703-beb7bac3f4d6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.380197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd852f-1dc1-46a1-a703-beb7bac3f4d6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.390117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdd852f-1dc1-46a1-a703-beb7bac3f4d6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:25.392184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdd852f-1dc1-46a1-a703-beb7bac3f4d6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-bfdd852f-1dc1-46a1-a703-beb7bac3f4d6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:25.408148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:25.410223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 282/550] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42553d1a-2a38-4291-ac6a-ea90de349376 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.412037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42553d1a-2a38-4291-ac6a-ea90de349376 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.412874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42553d1a-2a38-4291-ac6a-ea90de349376 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.413521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 283/551] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:25.415510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42553d1a-2a38-4291-ac6a-ea90de349376 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:25.439669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 269/552] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 525 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 24 08:38:25.441552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699fc7ff-2735-4bac-89f1-c55ba2580271 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.442080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699fc7ff-2735-4bac-89f1-c55ba2580271 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.442583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699fc7ff-2735-4bac-89f1-c55ba2580271 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.466151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-699fc7ff-2735-4bac-89f1-c55ba2580271 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:25.468077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-699fc7ff-2735-4bac-89f1-c55ba2580271 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:25.489428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:25.556729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 270/553] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:25.558379 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699fc7ff-2735-4bac-89f1-c55ba2580271 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:25.591025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42553d1a-2a38-4291-ac6a-ea90de349376 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:25.646048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:25.646764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.647106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.647442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.648295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:25.648717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:25.650131 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:25.663879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 284/554] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:25.664656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:25.685870 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:25.704106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699fc7ff-2735-4bac-89f1-c55ba2580271 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:25.748762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b817617-6d45-464c-b1d8-8ff507db35bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:25.749534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b817617-6d45-464c-b1d8-8ff507db35bf None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.749889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b817617-6d45-464c-b1d8-8ff507db35bf None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.750160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b817617-6d45-464c-b1d8-8ff507db35bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.764269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b817617-6d45-464c-b1d8-8ff507db35bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:25.766556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b817617-6d45-464c-b1d8-8ff507db35bf None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:25.782293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-8b817617-6d45-464c-b1d8-8ff507db35bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:25.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:25.795814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:25.796988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:25.796988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:25.796988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:25.796988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 271/555] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:25.797957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcaed78-6b8e-41bf-a128-4e123fa6b6f2 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:25.804819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c42461-2794-48cb-8889-7b79a7e31892 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.805177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c42461-2794-48cb-8889-7b79a7e31892 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.805430 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c42461-2794-48cb-8889-7b79a7e31892 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.807416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 272/556] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:25 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:25.808046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c42461-2794-48cb-8889-7b79a7e31892 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:25.842243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fba5f0-71c1-4356-8034-bd41877c2388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:25.842856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fba5f0-71c1-4356-8034-bd41877c2388 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:25.843134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fba5f0-71c1-4356-8034-bd41877c2388 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:25.843403 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fba5f0-71c1-4356-8034-bd41877c2388 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:25.844152 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fba5f0-71c1-4356-8034-bd41877c2388 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:25.845527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fba5f0-71c1-4356-8034-bd41877c2388 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:25.862405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 285/557] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:38:25 2022] POST /identity/v3/services => generated 240 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:25.863598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fba5f0-71c1-4356-8034-bd41877c2388 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:25.967516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c42461-2794-48cb-8889-7b79a7e31892 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:25.985214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:26.023730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88287bcd-7f65-4ff5-8662-b7a25aae97a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:26.024782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88287bcd-7f65-4ff5-8662-b7a25aae97a1 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:26.025214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88287bcd-7f65-4ff5-8662-b7a25aae97a1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:26.026144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88287bcd-7f65-4ff5-8662-b7a25aae97a1 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/128903595c5e454c8c51c371d014eb49/roles/a1265c18dedb42d5943905f144e48fb7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:26.027745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fba5f0-71c1-4356-8034-bd41877c2388 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:26.046737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88287bcd-7f65-4ff5-8662-b7a25aae97a1 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=128903595c5e454c8c51c371d014eb49, role_id=a1265c18dedb42d5943905f144e48fb7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:26.048296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88287bcd-7f65-4ff5-8662-b7a25aae97a1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:26.065357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 273/558] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:25 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/128903595c5e454c8c51c371d014eb49/roles/a1265c18dedb42d5943905f144e48fb7 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:26.066723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbcefb-d7a3-4198-90d6-5eb24ad1a0d4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:26.067007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbcefb-d7a3-4198-90d6-5eb24ad1a0d4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:26.067278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbcefb-d7a3-4198-90d6-5eb24ad1a0d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:26.095363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-64fbcefb-d7a3-4198-90d6-5eb24ad1a0d4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:26.099493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-64fbcefb-d7a3-4198-90d6-5eb24ad1a0d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:26.105334 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:26.106162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:26.106503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:26.106869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:26.107813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:26.108117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:26.109692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:26.128659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 286/559] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:26.129151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:26.190564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:26.236596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 274/560] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:26 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 24 08:38:26.237753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fbcefb-d7a3-4198-90d6-5eb24ad1a0d4 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:26.302393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f220a970-c51c-420e-9710-b388f7402332 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:26.349863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7ae728-3aa9-4b18-be48-f7c4db71789b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:26.350845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7ae728-3aa9-4b18-be48-f7c4db71789b None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:26.351328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7ae728-3aa9-4b18-be48-f7c4db71789b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:26.351794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7ae728-3aa9-4b18-be48-f7c4db71789b None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:26.352856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7ae728-3aa9-4b18-be48-f7c4db71789b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:26.355736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7ae728-3aa9-4b18-be48-f7c4db71789b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:26.359830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:26.381374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 287/561] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:38:26 2022] POST /identity/v3/services => generated 251 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:26.388641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fbcefb-d7a3-4198-90d6-5eb24ad1a0d4 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:26.441913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:26.442641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:26.442917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:26.443189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:26.451651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:26.453303 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:26.467096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:26.468080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:26.469009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 275/562] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:26.474461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7ae728-3aa9-4b18-be48-f7c4db71789b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:26.596399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7ae728-3aa9-4b18-be48-f7c4db71789b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:26.638378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:26.638998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:26.639281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:26.639550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:26.640368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:26.640677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:26.642149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:26.659933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 288/563] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:26.668991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:26.701446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:26.722568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:26.779634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:26.827449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6952d5c-ad59-4057-b6b7-d9f03f99f246 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:26.839926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20027c4-f7c0-405f-8d7e-33f3602959bc None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:26.883092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:26.883824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:26.884078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:26.884386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:26.885762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=4244858dfb464d029b6f6bed0491557d) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:26.886075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:26.887753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:26.888772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4801be39-7a7a-4d13-b086-7252717e4862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:26.889661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4801be39-7a7a-4d13-b086-7252717e4862 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:26.890374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4801be39-7a7a-4d13-b086-7252717e4862 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:26.890872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4801be39-7a7a-4d13-b086-7252717e4862 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:26.894949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 276/564] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:26 2022] GET /identity/v3/role_assignments?user.id=4244858dfb464d029b6f6bed0491557d&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 608 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:26.905486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4801be39-7a7a-4d13-b086-7252717e4862 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-4801be39-7a7a-4d13-b086-7252717e4862 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:26.911173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:26.912250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:38:26.913108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:26.916610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 289/565] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:38:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:26.923646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:27.090372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:27.098784 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe78bf3-969f-4e07-ac72-36d1a515ad20 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:27.150154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:27.151455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:27.151799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:27.152117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:27.152783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:27.153092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:27.154487 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:27.165105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 277/566] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:27.172865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4801be39-7a7a-4d13-b086-7252717e4862 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:27.318216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4801be39-7a7a-4d13-b086-7252717e4862 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:27.337169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:27.364427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:27.365571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:27.366117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:27.366708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:27.381310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:27.383454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:27.402573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:27.403560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:27.404744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:27.404744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:27.404744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:27.407249 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 290/567] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:27.413852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:27.583961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17725824-08da-4eb4-b99e-600be8e5359f None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:27.637318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:27.637939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:27.638235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:27.638520 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:27.639230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:27.639534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:27.640969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:27.662558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 278/568] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:27.669815 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:27.809974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6372102-71f5-4811-9095-437e7b34a789 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:27.849857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:27.850922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:27.851415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:27.851914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] PATH_INFO: `/v3/users/swift` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:27.862408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:27.864448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:27.879736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:27.881288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 24 08:38:27.882359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:27.885636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 291/569] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:38:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:27.892236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:27.914619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:28.043980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c00bd2b-64ad-4534-ab58-d5ad674c7e47 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:28.096442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:28.097052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:28.097329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:28.097594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:28.098322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:28.098619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:28.100119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:28.133320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 279/570] 10.222.0.39 () {56 vars in 1135 bytes} [Mon Oct 24 08:38:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:28.144267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:28.215103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:28.351493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b0875e-4553-4070-b1ed-1e34815b3483 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:28.394426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:28.395063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:28.395338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:28.395605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:28.408831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:28.410467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:28.428643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:28.429748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:28.430699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:28.430699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:28.430699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:28.437521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:28.438442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 292/571] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:28.564785 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:28.596494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ebcc1b-f8be-42bf-8c8f-89f03d5b11a0 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:28.641848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435e69d-533e-4162-b4fb-6f24afcac27d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:28.642458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e69d-533e-4162-b4fb-6f24afcac27d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:28.642736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e69d-533e-4162-b4fb-6f24afcac27d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:28.643006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e69d-533e-4162-b4fb-6f24afcac27d None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:28.643647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435e69d-533e-4162-b4fb-6f24afcac27d None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:28.643949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435e69d-533e-4162-b4fb-6f24afcac27d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:28.645358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435e69d-533e-4162-b4fb-6f24afcac27d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:28.662682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 280/572] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:28.668786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:28.828654 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:28.874415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d15d360-6a40-4c7f-8660-2997d84846aa None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:28.917383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee151e5-c07c-4b05-be76-cc00e462bdf8 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:28.917717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee151e5-c07c-4b05-be76-cc00e462bdf8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:28.917953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee151e5-c07c-4b05-be76-cc00e462bdf8 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:28.919591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 281/573] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:28.920179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:28.921103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:28.921576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:28.922056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:28.925732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaa9dbf-0908-4c42-8ef9-228770509f9d None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:28.926038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaa9dbf-0908-4c42-8ef9-228770509f9d None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:28.926292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaa9dbf-0908-4c42-8ef9-228770509f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:28.932212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:28.934371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:28.950967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:28.951939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:28.953037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 293/574] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:28.966950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:28.971303 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-dbaa9dbf-0908-4c42-8ef9-228770509f9d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:28.972811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-dbaa9dbf-0908-4c42-8ef9-228770509f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:29.080944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 282/575] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:29.244222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a500db-2f97-451e-b488-1bf1d63a8d4d None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:29.244948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a500db-2f97-451e-b488-1bf1d63a8d4d None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:29.245412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a500db-2f97-451e-b488-1bf1d63a8d4d None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:29.248729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 283/576] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:29.253608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427f21e4-a658-4669-9511-6daf8005d301 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:29.254210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427f21e4-a658-4669-9511-6daf8005d301 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:29.254715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427f21e4-a658-4669-9511-6daf8005d301 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:29.260347 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65364136-5c9f-4a6b-83b3-936220a9fb9a None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:29.276762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-427f21e4-a658-4669-9511-6daf8005d301 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:29.278442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-427f21e4-a658-4669-9511-6daf8005d301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:29.338952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:29.339910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:29.340593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:29.341816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:29.342738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:29.343456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:29.349495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:29.362613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:29.374698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 294/577] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 408 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:29.381430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:29.397724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:29.411994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 284/578] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:29 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:29.413399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427f21e4-a658-4669-9511-6daf8005d301 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:29.571463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce309050-e311-4290-b93d-8407f4e61020 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:29.596109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427f21e4-a658-4669-9511-6daf8005d301 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:29.688934 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cd261d-1b0d-4758-9a0d-48bf6a3dbf47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:29.690654 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cd261d-1b0d-4758-9a0d-48bf6a3dbf47 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:29.691143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cd261d-1b0d-4758-9a0d-48bf6a3dbf47 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:29.691640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cd261d-1b0d-4758-9a0d-48bf6a3dbf47 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/60a07d37339f4f83ad2ce8be3d0892a8/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:29.697594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cd261d-1b0d-4758-9a0d-48bf6a3dbf47 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=60a07d37339f4f83ad2ce8be3d0892a8, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:29.701635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5a7e0-5500-4fea-a3e4-4073e6ab7acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:29.702678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5a7e0-5500-4fea-a3e4-4073e6ab7acc None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:29.703174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5a7e0-5500-4fea-a3e4-4073e6ab7acc None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:29.703660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5a7e0-5500-4fea-a3e4-4073e6ab7acc None admin] PATH_INFO: `/v3/roles/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:29.705319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cd261d-1b0d-4758-9a0d-48bf6a3dbf47 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:29.739199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 295/579] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:29 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/60a07d37339f4f83ad2ce8be3d0892a8/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:29.740334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f5a7e0-5500-4fea-a3e4-4073e6ab7acc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:29.746284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cd261d-1b0d-4758-9a0d-48bf6a3dbf47 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-18f5a7e0-5500-4fea-a3e4-4073e6ab7acc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:29.748507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:29.749619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:29.751305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 285/580] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:38:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:29.757069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5a7e0-5500-4fea-a3e4-4073e6ab7acc None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:29.850635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:29.880457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:29.949685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cd261d-1b0d-4758-9a0d-48bf6a3dbf47 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:29.964934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5a7e0-5500-4fea-a3e4-4073e6ab7acc None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:30.018555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8693165c-c689-4e8a-923d-afd890ab74b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.019204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693165c-c689-4e8a-923d-afd890ab74b2 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.019488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693165c-c689-4e8a-923d-afd890ab74b2 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.019758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693165c-c689-4e8a-923d-afd890ab74b2 None admin] PATH_INFO: `/v3/services/network` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.020384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8693165c-c689-4e8a-923d-afd890ab74b2 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.021764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8693165c-c689-4e8a-923d-afd890ab74b2 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.028766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e186e0-25dd-4749-89aa-3ea41114b536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.029578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e186e0-25dd-4749-89aa-3ea41114b536 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.030088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e186e0-25dd-4749-89aa-3ea41114b536 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.030395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e186e0-25dd-4749-89aa-3ea41114b536 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.031418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e186e0-25dd-4749-89aa-3ea41114b536 None admin] RBAC: Adding query filter params (name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-8693165c-c689-4e8a-923d-afd890ab74b2 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:30.031595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:30.032733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:30.033716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:30.035188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e186e0-25dd-4749-89aa-3ea41114b536 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.035264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 296/581] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:29 2022] GET /identity/v3/services/network => generated 88 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:30.035332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e186e0-25dd-4749-89aa-3ea41114b536 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.035445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc1c2c8-d281-4b2e-8370-e3b00d35ad66 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.035445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc1c2c8-d281-4b2e-8370-e3b00d35ad66 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.035445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc1c2c8-d281-4b2e-8370-e3b00d35ad66 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.041565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 297/582] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:30.041565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27675ff1-5d46-4feb-9163-be78ac45b3fc None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.041565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27675ff1-5d46-4feb-9163-be78ac45b3fc None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.041565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27675ff1-5d46-4feb-9163-be78ac45b3fc None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.041565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 298/583] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:30 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:30.041565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27675ff1-5d46-4feb-9163-be78ac45b3fc None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:30.053899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 286/584] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:38:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:30.056676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2becab-7d19-4642-b0b4-161d530fa251 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.056676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2becab-7d19-4642-b0b4-161d530fa251 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.056676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2becab-7d19-4642-b0b4-161d530fa251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.077040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7f2becab-7d19-4642-b0b4-161d530fa251 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:30.078693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7f2becab-7d19-4642-b0b4-161d530fa251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:30.172162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 287/585] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:30.172162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09b9ce-b456-4277-a85e-1d47b5ad115c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.172162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09b9ce-b456-4277-a85e-1d47b5ad115c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.172162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09b9ce-b456-4277-a85e-1d47b5ad115c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.194549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27675ff1-5d46-4feb-9163-be78ac45b3fc None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:30.196543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-da09b9ce-b456-4277-a85e-1d47b5ad115c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:30.198372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-da09b9ce-b456-4277-a85e-1d47b5ad115c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:30.236777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:30.267398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.268189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.268577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.268954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.269686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] RBAC: Adding query filter params (name=network) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:30.270111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.270261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:30.272457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.279333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 299/586] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/services?name=network => generated 124 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:30.281035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:30.306180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 288/587] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:30.308540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09b9ce-b456-4277-a85e-1d47b5ad115c None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:30.423905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09b9ce-b456-4277-a85e-1d47b5ad115c None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:30.434183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b0da66-5bc1-4fec-a1b0-d91d3b74de2a None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:30.484432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.485114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.485445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.485766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.501735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.502798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.503544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.504122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.504695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.505191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.506038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] RBAC: Adding query filter params (type=network) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:30.506559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.508191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.520602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 289/588] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/services?type=network => generated 350 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:30.522715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:30.523655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:30.524596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:30.524596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:30.524596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:30.524596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 300/589] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:30.527437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:30.534313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:30.579049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:30.589027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:30.644937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabb82d-08c1-4d05-a4dd-143f67b09eca None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:30.649664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91da4d46-6f40-477c-b1d8-387b372b8313 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:30.684208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.684754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.685359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.685496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.686097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] RBAC: Adding query filter params (interface=public, service_id=17fa13f54ae54704a0def32327930111, region_id=RegionOne) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:30.686421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.687809 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.691095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.691969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.692489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.693050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.693897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:30.694405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.696106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.696388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 290/590] 10.222.0.39 () {56 vars in 1251 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/endpoints?service_id=17fa13f54ae54704a0def32327930111&interface=public®ion_id=RegionOne => generated 194 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:30.697217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:30.711479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 301/591] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:30.712770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:30.821389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27b3ef3-e78f-4ecc-8330-5243959fd681 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:30.832283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:30.847513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:30.851431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93f4d7-5b6e-40d5-b857-ffd0e64e302f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:30.876369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.876944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.877275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.877492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.878072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.880291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:30.889529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:30.890671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:30.891595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 291/592] 10.222.0.39 () {56 vars in 1129 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:30.892822 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:30.911162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677f6bc2-2a72-4334-828d-765f24271f47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:30.911653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677f6bc2-2a72-4334-828d-765f24271f47 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:30.911969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677f6bc2-2a72-4334-828d-765f24271f47 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:30.912391 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677f6bc2-2a72-4334-828d-765f24271f47 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:30.913148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677f6bc2-2a72-4334-828d-765f24271f47 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:30.914825 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677f6bc2-2a72-4334-828d-765f24271f47 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:30.923879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 302/593] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/services => generated 772 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:30.925326 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677f6bc2-2a72-4334-828d-765f24271f47 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.090688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9cf504-8c0f-4ed4-b30a-a892bcda6e26 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.107456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677f6bc2-2a72-4334-828d-765f24271f47 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.149178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.149993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.151088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.151224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.151928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:31.153480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:31.169521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:31.170545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:31.171448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:31.172265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.172265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.172265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.172482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 292/594] 10.222.0.39 () {56 vars in 1132 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:31.172554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.174240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.186760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:31.188293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:31.200874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:31.201904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:31.202849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 303/595] 10.222.0.39 () {56 vars in 1117 bytes} [Mon Oct 24 08:38:30 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:31.208527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.234310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:31.252771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:31.336549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd5cdb-3acd-49b1-8258-aa063c1f28d5 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.349343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d27076c-4fe8-4f95-8219-2084cf8b30ff None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.382905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.384230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.384727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.385731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.386668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:31.387511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:31.389944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:31.394616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.395314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.395779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.400316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.400316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:31.400316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:31.400316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:31.401114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 293/596] 10.222.0.39 () {56 vars in 1124 bytes} [Mon Oct 24 08:38:31 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:31.403043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.406443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 304/597] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:31 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:31.409548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.538561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:31.553373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:31.555749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56118b5-7b99-4249-95ba-b4ead66ec8f9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.559663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82e10ae-e20d-4e15-9b9a-d2632dae4a97 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.599582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.600656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.600767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.601255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.601594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.601908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.602078 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.602495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.603189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:31.603297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:31.604195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:31.604573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:31.606175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:31.606427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:31.616216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 305/598] 10.222.0.39 () {56 vars in 1124 bytes} [Mon Oct 24 08:38:31 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:31.616216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.641221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 294/599] 10.222.0.39 () {56 vars in 1149 bytes} [Mon Oct 24 08:38:31 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:31.642694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.729321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4890b800-424c-443a-86a0-def04076e9f9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.766935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.767645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.767995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.768380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.769112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:31.769659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:31.771454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:31.783417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 306/600] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:31 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:31.784884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.803985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:31.804791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3678e8c2-6b46-4721-b07a-cadbcd3fdc37 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.899201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.899840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.900188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.900936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.901876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] RBAC: Adding query filter params (interface=public, service_id=f966775317a94b73ac79e2c2cc02156e, region_id=RegionOne) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:31.902174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:31.903790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:31.914111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cddab3-2afe-4aca-9865-be434bbfff97 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:31.918643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 295/601] 10.222.0.39 () {56 vars in 1251 bytes} [Mon Oct 24 08:38:31 2022] GET /identity/v3/endpoints?service_id=f966775317a94b73ac79e2c2cc02156e&interface=public®ion_id=RegionOne => generated 194 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:31.924436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:31.960433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5622aa-54ce-4c86-9577-a92b3ebc985c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:31.961073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5622aa-54ce-4c86-9577-a92b3ebc985c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:31.961350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5622aa-54ce-4c86-9577-a92b3ebc985c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:31.961624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5622aa-54ce-4c86-9577-a92b3ebc985c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:31.982391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:32.124642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5622aa-54ce-4c86-9577-a92b3ebc985c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:32.126455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5622aa-54ce-4c86-9577-a92b3ebc985c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:32.157795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eefa651-fa26-4031-8d85-671be49a4f0d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:32.158026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:32.237695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98db27f-69e5-47ad-9df4-8e8d353fac63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:32.238446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98db27f-69e5-47ad-9df4-8e8d353fac63 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:32.238804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98db27f-69e5-47ad-9df4-8e8d353fac63 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:32.239075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98db27f-69e5-47ad-9df4-8e8d353fac63 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:32.239817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98db27f-69e5-47ad-9df4-8e8d353fac63 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:32.243043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98db27f-69e5-47ad-9df4-8e8d353fac63 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:32.251047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 296/602] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:38:31 2022] GET /identity/v3/services => generated 772 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:32.256995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e951a-5740-4c25-8566-a1544a16ab94 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:32.257338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e951a-5740-4c25-8566-a1544a16ab94 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:32.257614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e951a-5740-4c25-8566-a1544a16ab94 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:32.259472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 297/603] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:32 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:32.268922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d408b20-702a-4ec5-a387-35ff670f0c47 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:32.269319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d408b20-702a-4ec5-a387-35ff670f0c47 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:32.269594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d408b20-702a-4ec5-a387-35ff670f0c47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:32.309561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3d408b20-702a-4ec5-a387-35ff670f0c47 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:32.310851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3d408b20-702a-4ec5-a387-35ff670f0c47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:32.316960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 307/604] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:31 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 533 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 24 08:38:32.325479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5622aa-54ce-4c86-9577-a92b3ebc985c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:32.473251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 298/605] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:32.535441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5622aa-54ce-4c86-9577-a92b3ebc985c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:32.555384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:32.587041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:32.587717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:32.588928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:32.589206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:32.600832 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:32.602239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:32.621054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:32.622105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:32.623350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:32.623350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:32.623350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:32.623350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 308/606] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:32.627862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d408b20-702a-4ec5-a387-35ff670f0c47 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:32.647370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:32.762163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d408b20-702a-4ec5-a387-35ff670f0c47 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:32.811791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:32.812621 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:32.812808 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:32.812966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:32.813578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:32.813942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:32.815370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:32.835567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 299/607] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:32.838541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:32.846566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:32.964610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:33.089509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aaa4d5-c22f-45cc-bdd6-e14d29f54860 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:33.174996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356671f0-3105-4b59-a33c-e555a8dbc536 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:33.206876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:33.207522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:33.207801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:33.208077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:33.216587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b79037a-af57-4dd7-bfb6-41c16609d492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:33.217656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b79037a-af57-4dd7-bfb6-41c16609d492 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:33.218254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b79037a-af57-4dd7-bfb6-41c16609d492 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:33.218767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b79037a-af57-4dd7-bfb6-41c16609d492 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:33.220711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:33.222316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:33.233029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b79037a-af57-4dd7-bfb6-41c16609d492 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-6b79037a-af57-4dd7-bfb6-41c16609d492 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:33.239720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:33.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:38:33.241776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:33.245404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 309/608] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:38:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 407 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:33.250898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:33.252459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:33.253592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:33.253592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:33.253592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:33.253592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:33.253592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:33.253592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:33.253592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:33.253592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:33.255701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b79037a-af57-4dd7-bfb6-41c16609d492 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:33.258592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 300/609] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 413 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:33.265120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:33.338711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:33.454478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b79037a-af57-4dd7-bfb6-41c16609d492 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:33.496310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:33.497384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:33.497928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:33.498398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:33.499268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:33.499786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:33.501383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:33.505704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2f60ea-b43f-4f3b-861a-373e4aac359c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:33.512276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 310/610] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:33.523044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:33.621019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:33.621573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:33.621853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:33.622123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:33.622830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:33.623134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:33.624636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:33.635295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:33.638180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 301/611] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 374 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:33.638703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:33.644095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:33.653939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8510fa9-a07b-4f35-bef4-ce9e6a18ab8d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:33.697914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:33.699174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:33.699788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:33.700412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:33.714294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:33.716339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:33.734488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:33.735536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:33.736487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:33.736487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:33.736487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:33.738673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 311/612] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:33.745826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:33.875441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598a3924-3f70-4b7d-9af5-e0feb7c8cab5 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:33.921608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8102a2ad-f8d6-482a-9594-ba99271e16c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:33.923106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8102a2ad-f8d6-482a-9594-ba99271e16c8 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:33.923836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8102a2ad-f8d6-482a-9594-ba99271e16c8 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:33.924314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8102a2ad-f8d6-482a-9594-ba99271e16c8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:33.925474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8102a2ad-f8d6-482a-9594-ba99271e16c8 None admin] RBAC: Adding query filter params (role.id=a1265c18dedb42d5943905f144e48fb7, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=128903595c5e454c8c51c371d014eb49) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:33.926536 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8102a2ad-f8d6-482a-9594-ba99271e16c8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:33.928667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8102a2ad-f8d6-482a-9594-ba99271e16c8 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:33.939219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 302/613] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:33 2022] GET /identity/v3/role_assignments?user.id=128903595c5e454c8c51c371d014eb49&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=a1265c18dedb42d5943905f144e48fb7 => generated 608 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:33.940905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e199a4f4-fa11-4376-9785-1b916b014a83 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:33.941772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e199a4f4-fa11-4376-9785-1b916b014a83 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:33.942510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e199a4f4-fa11-4376-9785-1b916b014a83 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:33.945099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 303/614] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:33.955996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff216d14-1abe-46a6-b939-78f36b378c19 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:33.956741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff216d14-1abe-46a6-b939-78f36b378c19 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:33.957306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff216d14-1abe-46a6-b939-78f36b378c19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:33.964074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6b6baa-ed39-4ced-bb00-4bf6663f23e9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:33.997371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-ff216d14-1abe-46a6-b939-78f36b378c19 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:33.998631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ff216d14-1abe-46a6-b939-78f36b378c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:34.015303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:34.016242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:34.016354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:34.016651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:34.017341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:34.017642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:34.019186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:34.030960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:34.036679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 312/615] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:34.045526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:34.115962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:34.156188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 304/616] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:33 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:34.250457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6db908-facf-4fe7-9692-c22fabf3fa47 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:34.310227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c17254-77e4-4245-826a-438b0939752f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:34.311267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c17254-77e4-4245-826a-438b0939752f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:34.311744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c17254-77e4-4245-826a-438b0939752f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:34.312211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c17254-77e4-4245-826a-438b0939752f None admin] PATH_INFO: `/v3/users/swift` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:34.326434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c17254-77e4-4245-826a-438b0939752f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:34.328959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c17254-77e4-4245-826a-438b0939752f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-08c17254-77e4-4245-826a-438b0939752f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:34.346240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:34.347242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:34.348242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 313/617] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:38:34 2022] GET /identity/v3/users/swift => generated 83 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:34.357339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff216d14-1abe-46a6-b939-78f36b378c19 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:34.502561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c17254-77e4-4245-826a-438b0939752f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:34.551697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff216d14-1abe-46a6-b939-78f36b378c19 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:34.627035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:34.627971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:34.628691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:34.629220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:34.631362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:34.632051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:34.635473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:34.646311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:34.677647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 305/618] 10.222.0.39 () {56 vars in 1135 bytes} [Mon Oct 24 08:38:34 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:34.686670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:34.738391 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c17254-77e4-4245-826a-438b0939752f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:34.774674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:34.815039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:34.815686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:34.815964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:34.816265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] PATH_INFO: `/v3/services/network` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:34.816896 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:34.818426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:34.829726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:34.830867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 24 08:38:34.831728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:34.836470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 314/619] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:34 2022] GET /identity/v3/services/network => generated 88 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:34.839013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:34.866465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc054e-9080-420c-b755-86a4d9009a24 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:34.923928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d07c2b8-6150-4bef-a38f-3be50cf4a6bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:34.925026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d07c2b8-6150-4bef-a38f-3be50cf4a6bc None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:34.925481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d07c2b8-6150-4bef-a38f-3be50cf4a6bc None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:34.925846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d07c2b8-6150-4bef-a38f-3be50cf4a6bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.011635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925bda4b-ea5b-4c95-b552-f88d1a24e052 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:35.043449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:35.057394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1351cc-ffdf-4dee-bbac-32e29c2741a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:35.058075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1351cc-ffdf-4dee-bbac-32e29c2741a0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.058354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1351cc-ffdf-4dee-bbac-32e29c2741a0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.058626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1351cc-ffdf-4dee-bbac-32e29c2741a0 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.059264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1351cc-ffdf-4dee-bbac-32e29c2741a0 None admin] RBAC: Adding query filter params (name=network) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:35.059562 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1351cc-ffdf-4dee-bbac-32e29c2741a0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:35.060992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1351cc-ffdf-4dee-bbac-32e29c2741a0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:35.062727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d07c2b8-6150-4bef-a38f-3be50cf4a6bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:35.065737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d07c2b8-6150-4bef-a38f-3be50cf4a6bc None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:35.066967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 315/620] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:34 2022] GET /identity/v3/services?name=network => generated 124 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:35.068393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d4364-9fe6-42ed-b5c8-2d75b20de31b None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.068689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d4364-9fe6-42ed-b5c8-2d75b20de31b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.068974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d4364-9fe6-42ed-b5c8-2d75b20de31b None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.075897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 316/621] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:35 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:35.077014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95d4364-9fe6-42ed-b5c8-2d75b20de31b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:35.141768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:35.202905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95d4364-9fe6-42ed-b5c8-2d75b20de31b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:35.247688 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:35.248636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.249061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.249519 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.250372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] RBAC: Adding query filter params (type=network) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:35.250826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:35.252492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:35.252819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 306/622] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:34 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 566 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 24 08:38:35.255649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb080f35-fc88-455c-a840-c9bca13bc222 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.256005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb080f35-fc88-455c-a840-c9bca13bc222 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.256615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb080f35-fc88-455c-a840-c9bca13bc222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.259633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 317/623] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:35 2022] GET /identity/v3/services?type=network => generated 350 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:35.261256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:35.289392 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-fb080f35-fc88-455c-a840-c9bca13bc222 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:35.291939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-fb080f35-fc88-455c-a840-c9bca13bc222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:35.323613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:35.381607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d9d98-907a-4bf7-a2c1-6e0e0a09bc5f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:35.428159 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 307/624] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:35 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:35.428159 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb080f35-fc88-455c-a840-c9bca13bc222 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:35.432997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33ae0f0-ca9e-482f-983e-5df445fed206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:35.433993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33ae0f0-ca9e-482f-983e-5df445fed206 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.434358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33ae0f0-ca9e-482f-983e-5df445fed206 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.434667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33ae0f0-ca9e-482f-983e-5df445fed206 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.449042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33ae0f0-ca9e-482f-983e-5df445fed206 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:35.450801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33ae0f0-ca9e-482f-983e-5df445fed206 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-f33ae0f0-ca9e-482f-983e-5df445fed206 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:35.469746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:35.470779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:35.471699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:35.471699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:35.471699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:35.476561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 318/625] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:35.478290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb07b1-a44a-40be-a652-50a0ddcdcba5 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.479454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb07b1-a44a-40be-a652-50a0ddcdcba5 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.479741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb07b1-a44a-40be-a652-50a0ddcdcba5 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.482011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 319/626] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:35.482811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cb07b1-a44a-40be-a652-50a0ddcdcba5 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:35.533351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:35.569456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb080f35-fc88-455c-a840-c9bca13bc222 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:35.618851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cb07b1-a44a-40be-a652-50a0ddcdcba5 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:35.622597 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:35.625907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206bda45-e322-463b-833e-e1ff137b4919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:35.626822 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206bda45-e322-463b-833e-e1ff137b4919 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.627332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206bda45-e322-463b-833e-e1ff137b4919 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.627959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206bda45-e322-463b-833e-e1ff137b4919 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.630507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206bda45-e322-463b-833e-e1ff137b4919 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:35.632927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206bda45-e322-463b-833e-e1ff137b4919 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:35.668387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 308/627] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:38:35 2022] POST /identity/v3/endpoints => generated 337 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:35.670768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0add3ff7-a439-48eb-8a4f-fd5e362c37aa None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.671338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0add3ff7-a439-48eb-8a4f-fd5e362c37aa None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.672021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0add3ff7-a439-48eb-8a4f-fd5e362c37aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.686464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:35.687337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.687493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.687833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.688577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:35.689007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:35.691256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:35.707505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-0add3ff7-a439-48eb-8a4f-fd5e362c37aa None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:35.709631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0add3ff7-a439-48eb-8a4f-fd5e362c37aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:35.724592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 320/628] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:35.735112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:35.828617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 309/629] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:35 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:35.829984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0add3ff7-a439-48eb-8a4f-fd5e362c37aa None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:35.865148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b4b725-6096-4ec5-bf8d-ee32eec10640 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:35.912617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:35.913222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:35.913502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:35.913808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:35.925658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:35.927693 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:35.945746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:35.947700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 321/630] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:35.955213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:35.980248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:36.003629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:36.053810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0add3ff7-a439-48eb-8a4f-fd5e362c37aa None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:36.099931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dd1484-ee30-4fc1-b609-2c1e5e7aa8a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:36.100590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dd1484-ee30-4fc1-b609-2c1e5e7aa8a2 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:36.100874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dd1484-ee30-4fc1-b609-2c1e5e7aa8a2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:36.101142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dd1484-ee30-4fc1-b609-2c1e5e7aa8a2 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:36.101911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dd1484-ee30-4fc1-b609-2c1e5e7aa8a2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:36.103264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3309f221-107c-4e46-ad1b-fcb1c3c99844 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:36.103383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dd1484-ee30-4fc1-b609-2c1e5e7aa8a2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:36.126612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 310/631] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:38:35 2022] POST /identity/v3/services => generated 269 bytes in 297 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:36.163062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:36.163716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:36.164055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:36.164377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:36.165151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:36.165464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:36.168154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:36.185302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 322/632] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:36.193810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dd1484-ee30-4fc1-b609-2c1e5e7aa8a2 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:36.227814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:36.367814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dd1484-ee30-4fc1-b609-2c1e5e7aa8a2 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:36.375751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:36.405420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:36.415306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dd9db5-6566-4757-9e2e-00d13694c611 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:36.432659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:36.433767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:36.434409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:36.434900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:36.435865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=60a07d37339f4f83ad2ce8be3d0892a8) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:36.436432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:36.439054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:36.451477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 311/633] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:36 2022] GET /identity/v3/role_assignments?user.id=60a07d37339f4f83ad2ce8be3d0892a8&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 608 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:36.482837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:36.483682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:36.484169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:36.484651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:36.485540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:36.487405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:36.501219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:36.503230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:36.504925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 323/634] 10.222.0.39 () {56 vars in 1129 bytes} [Mon Oct 24 08:38:36 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:36.513761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:36.648834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c990c77-a388-4212-a11b-4fdd1f0b3372 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:36.698905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b8f798-1e7e-431b-a160-76d7cb255143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:36.699434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b8f798-1e7e-431b-a160-76d7cb255143 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:36.699711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b8f798-1e7e-431b-a160-76d7cb255143 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:36.699986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b8f798-1e7e-431b-a160-76d7cb255143 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:36.700649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b8f798-1e7e-431b-a160-76d7cb255143 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:36.700952 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b8f798-1e7e-431b-a160-76d7cb255143 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:36.702361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b8f798-1e7e-431b-a160-76d7cb255143 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:36.708331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 312/635] 10.222.0.39 () {56 vars in 1124 bytes} [Mon Oct 24 08:38:36 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:36.716067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:36.855575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c26e95-03bf-4311-9c37-3424df4ed6aa None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:36.904636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:36.905629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:36.905963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:36.906267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:36.906999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:36.907336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:36.909125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:36.910570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5bf276-1e45-4790-9fad-93bd77091ce5 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:36.910956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5bf276-1e45-4790-9fad-93bd77091ce5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:36.911321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5bf276-1e45-4790-9fad-93bd77091ce5 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:36.913527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 313/636] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:36.917473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 324/637] 10.222.0.39 () {56 vars in 1124 bytes} [Mon Oct 24 08:38:36 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:36.920487 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1618862-1f1f-4203-8fa9-b8a52a83357e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:36.920845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1618862-1f1f-4203-8fa9-b8a52a83357e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:36.921120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1618862-1f1f-4203-8fa9-b8a52a83357e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:36.923503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:36.927765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:36.947139 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a1618862-1f1f-4203-8fa9-b8a52a83357e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:36.949782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a1618862-1f1f-4203-8fa9-b8a52a83357e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:37.027631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:37.061123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 314/638] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:36 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:37.097943 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce399d05-e65e-4080-965c-db7c7e474451 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:37.138092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81240351-cc5e-4fad-9e45-c547fcc2c732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:37.138712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81240351-cc5e-4fad-9e45-c547fcc2c732 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:37.138991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81240351-cc5e-4fad-9e45-c547fcc2c732 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:37.139263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81240351-cc5e-4fad-9e45-c547fcc2c732 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:37.139931 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81240351-cc5e-4fad-9e45-c547fcc2c732 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:37.141348 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81240351-cc5e-4fad-9e45-c547fcc2c732 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:37.167070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 325/639] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:38:36 2022] POST /identity/v3/endpoints => generated 346 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:38:37.381495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1618862-1f1f-4203-8fa9-b8a52a83357e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:37.509028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1618862-1f1f-4203-8fa9-b8a52a83357e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:37.559020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560b755a-8f42-4917-81d1-ecec69a4f8fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:37.559672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b755a-8f42-4917-81d1-ecec69a4f8fd None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:37.559952 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b755a-8f42-4917-81d1-ecec69a4f8fd None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:37.560260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b755a-8f42-4917-81d1-ecec69a4f8fd None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:37.573106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560b755a-8f42-4917-81d1-ecec69a4f8fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-560b755a-8f42-4917-81d1-ecec69a4f8fd None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:37.578144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:37.579140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:37.580021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 315/640] 10.222.0.39 () {56 vars in 1120 bytes} [Mon Oct 24 08:38:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:37.587026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81240351-cc5e-4fad-9e45-c547fcc2c732 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:37.669080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:37.716837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81240351-cc5e-4fad-9e45-c547fcc2c732 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:37.755223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:37.756328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:37.756817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:37.757308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:37.758207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:37.758515 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:37.759970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:37.769571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 326/641] 10.222.0.39 () {56 vars in 1115 bytes} [Mon Oct 24 08:38:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:37.777897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560b755a-8f42-4917-81d1-ecec69a4f8fd None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:37.828099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:37.930686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560b755a-8f42-4917-81d1-ecec69a4f8fd None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:37.998190 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:37.999239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:37.999725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:38.000326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:38.017675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:38.020217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:38.046869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:38.048417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:38.049699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:38.049699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:38.049699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:38.049699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 316/642] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:38.056782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:38.170563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e4800-00bd-4ca7-b74e-5dd3d684c70f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:38.196005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:38.209332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:38.210559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:38.210830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:38.211101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:38.211745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:38.212118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:38.213594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:38.227300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 327/643] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:38.233514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:38.339310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:38.354108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03d3a69-34ad-4132-8af8-1c089173175f None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:38.393090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:38.393733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:38.394013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:38.394287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:38.403080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:38.404566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:38.417998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:38.419051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:38.420046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 317/644] 10.222.0.39 () {56 vars in 1117 bytes} [Mon Oct 24 08:38:38 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:38.425488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:38.537317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712fe4e1-336d-407f-8206-69014bbbf28b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:38.581270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:38.582196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:38.582504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:38.582817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:38.583929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:38.584382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:38.586454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:38.616783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 328/645] 10.222.0.39 () {56 vars in 1149 bytes} [Mon Oct 24 08:38:38 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:38.623814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:38.739647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6208cbf-2c33-4af8-ba31-e1b8cbb95234 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:38.778853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac54119c-2c7b-4826-9310-9b88cfb22073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:38.779497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54119c-2c7b-4826-9310-9b88cfb22073 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:38.779774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54119c-2c7b-4826-9310-9b88cfb22073 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:38.780047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54119c-2c7b-4826-9310-9b88cfb22073 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:38.797738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac54119c-2c7b-4826-9310-9b88cfb22073 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:38.799188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac54119c-2c7b-4826-9310-9b88cfb22073 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:38.810556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-ac54119c-2c7b-4826-9310-9b88cfb22073 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:38.819602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:38.820698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:38.821600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:38.821600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:38.821600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:38.821600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 318/646] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:38.827508 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:38.878967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:38.950152 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9181c0-2873-4a80-869b-5b1a45cf259a None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:38.999798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:39.000508 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.000811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.001113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.001970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:39.002314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:39.003910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:39.008583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121287d8-480a-4c83-960c-eded49e32585 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.009026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121287d8-480a-4c83-960c-eded49e32585 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.009478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121287d8-480a-4c83-960c-eded49e32585 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.013896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 319/647] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:39 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:39.020021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 329/648] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:39.022451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79026e-89da-4f73-9239-41d80bb90145 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.022941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79026e-89da-4f73-9239-41d80bb90145 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.023383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79026e-89da-4f73-9239-41d80bb90145 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.027189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:39.055467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5b79026e-89da-4f73-9239-41d80bb90145 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:39.058866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5b79026e-89da-4f73-9239-41d80bb90145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:39.165621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:39.170021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c214179a-f59d-4f8a-bee7-be6310241a4a None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:39.175409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 320/649] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:39 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:39.207573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5a241a-916b-42ac-ab1a-aa588c9a13d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:39.208620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a241a-916b-42ac-ab1a-aa588c9a13d5 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.209109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a241a-916b-42ac-ab1a-aa588c9a13d5 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.209583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a241a-916b-42ac-ab1a-aa588c9a13d5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.218044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5a241a-916b-42ac-ab1a-aa588c9a13d5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:39.219870 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5a241a-916b-42ac-ab1a-aa588c9a13d5 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-ca5a241a-916b-42ac-ab1a-aa588c9a13d5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:39.233146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:39.234143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:39.235064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:39.235064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:39.235064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:39.235064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:39.235064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:39.235064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:39.235064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:39.235064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:39.237645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 330/650] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:39.245912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b79026e-89da-4f73-9239-41d80bb90145 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:39.252607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1c659-74f3-466c-88d2-68cfa3f45eb7 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.252958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1c659-74f3-466c-88d2-68cfa3f45eb7 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.253288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1c659-74f3-466c-88d2-68cfa3f45eb7 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.254786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 331/651] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:39.261875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268561e-abc9-4f56-b929-915c7125285d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.262277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268561e-abc9-4f56-b929-915c7125285d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.262557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268561e-abc9-4f56-b929-915c7125285d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.288248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f268561e-abc9-4f56-b929-915c7125285d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:39.290691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f268561e-abc9-4f56-b929-915c7125285d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:39.375863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b79026e-89da-4f73-9239-41d80bb90145 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:39.405229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 332/652] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:39 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:39.432723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:39.438022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:39.438869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.439216 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.439492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.440218 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:39.440523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:39.442008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:39.457160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 321/653] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:39.464100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268561e-abc9-4f56-b929-915c7125285d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:39.470357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:39.606138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ccfc15-3e04-47e1-aa51-9c698a937024 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:39.622677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:39.641350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268561e-abc9-4f56-b929-915c7125285d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:39.656967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a212f292-cfc3-4d50-aaf1-55bb2ece3ccb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:39.658068 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212f292-cfc3-4d50-aaf1-55bb2ece3ccb None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.658570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212f292-cfc3-4d50-aaf1-55bb2ece3ccb None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.659048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212f292-cfc3-4d50-aaf1-55bb2ece3ccb None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.659862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a212f292-cfc3-4d50-aaf1-55bb2ece3ccb None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:39.661718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a212f292-cfc3-4d50-aaf1-55bb2ece3ccb None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-a212f292-cfc3-4d50-aaf1-55bb2ece3ccb None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:39.672418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:39.673445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:39.674349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:39.677221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 322/654] 10.222.0.39 () {56 vars in 1132 bytes} [Mon Oct 24 08:38:39 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:39.678817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b38c9c-5948-4189-b59b-d764863871c5 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.679315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b38c9c-5948-4189-b59b-d764863871c5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.679791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b38c9c-5948-4189-b59b-d764863871c5 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.681586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 323/655] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:39.684653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d389810-cd9e-4b55-b94a-57ce202e6f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:39.685325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d389810-cd9e-4b55-b94a-57ce202e6f55 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.685596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d389810-cd9e-4b55-b94a-57ce202e6f55 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.685865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d389810-cd9e-4b55-b94a-57ce202e6f55 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/128903595c5e454c8c51c371d014eb49/roles/eb1426874ce540f2a0d2c7f30c3b87f6` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.686423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b38c9c-5948-4189-b59b-d764863871c5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:39.688293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d389810-cd9e-4b55-b94a-57ce202e6f55 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=128903595c5e454c8c51c371d014eb49, role_id=eb1426874ce540f2a0d2c7f30c3b87f6)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:39.690101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d389810-cd9e-4b55-b94a-57ce202e6f55 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:39.711909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 333/656] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:38:39 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/128903595c5e454c8c51c371d014eb49/roles/eb1426874ce540f2a0d2c7f30c3b87f6 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:38:39.713620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adb5ccf-e21b-48b8-87e5-1aa14c1bb411 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.713989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adb5ccf-e21b-48b8-87e5-1aa14c1bb411 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.714645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adb5ccf-e21b-48b8-87e5-1aa14c1bb411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.742018 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-1adb5ccf-e21b-48b8-87e5-1aa14c1bb411 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:39.744339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1adb5ccf-e21b-48b8-87e5-1aa14c1bb411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:39.780763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:39.839332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 334/657] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:39 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:39.840175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adb5ccf-e21b-48b8-87e5-1aa14c1bb411 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:39.872452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b38c9c-5948-4189-b59b-d764863871c5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:39.937565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:39.938139 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:39.938413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:39.938686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:39.939315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:39.939615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:39.941113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:39.946677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 324/658] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:39 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:39.947674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:39.954353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:39.988465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adb5ccf-e21b-48b8-87e5-1aa14c1bb411 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:40.053556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:40.054867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:40.055343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:40.055625 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:40.056474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:40.060190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:40.081344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:40.082474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:40.083496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:40.084868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 335/659] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:39 2022] GET /identity/v3/services/object-store => generated 93 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:40.092872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:40.094442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e439d7c-428d-4fc7-8ae5-9af2c1160a53 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:40.124378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:40.147137 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:40.148001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:40.148459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:40.148939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:40.149776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:40.150892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:40.152682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:40.159157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 325/660] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:39 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:40.164597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:40.217233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e911fea4-0287-4e6a-81af-38649da657f8 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:40.276846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:40.277934 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:40.278392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:40.278841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:40.279981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:40.280683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:40.283594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:40.291843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 336/661] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:40 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:40.293302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:40.306311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:40.347808 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd7e7fc-fdd5-48f6-bf3e-2a8fa99d7911 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:40.412036 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90dfef2-6c0c-4722-9224-96a57e43386d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:40.429700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:40.431210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:40.431833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:40.432455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:40.434163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] RBAC: Adding query filter params (interface=public, service_id=06595b439b28451895be1edc174e9b5b, region_id=RegionOne) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:40.434625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:40.436758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:40.455024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 326/662] 10.222.0.39 () {56 vars in 1251 bytes} [Mon Oct 24 08:38:40 2022] GET /identity/v3/endpoints?service_id=06595b439b28451895be1edc174e9b5b&interface=public®ion_id=RegionOne => generated 194 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:40.455024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:40.464615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:40.465272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:40.465554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:40.465830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:40.466428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:40.468196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:40.481301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:40.482651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:40.483587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 337/663] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:40 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:40.484861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:40.531646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:40.587418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:40.627358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50ca37a-4aa4-4790-a50a-096fcaefe566 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:40.635332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df2d5f9-cbb2-4789-85ec-6f0ebc27aaca None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:40.670954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b26f7db-0cee-49c2-96cc-b5b007d0cc52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:40.672187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b26f7db-0cee-49c2-96cc-b5b007d0cc52 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:40.672557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b26f7db-0cee-49c2-96cc-b5b007d0cc52 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:40.672865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b26f7db-0cee-49c2-96cc-b5b007d0cc52 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:40.673530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b26f7db-0cee-49c2-96cc-b5b007d0cc52 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:40.675063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b26f7db-0cee-49c2-96cc-b5b007d0cc52 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:40.677201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:40.677824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:40.678122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:40.678399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:40.679082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:40.679325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:40.680795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:40.683299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 338/664] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:38:40 2022] GET /identity/v3/services => generated 1029 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:38:40.683568 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b26f7db-0cee-49c2-96cc-b5b007d0cc52 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:40.696170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 327/665] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:40 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:40.862732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b26f7db-0cee-49c2-96cc-b5b007d0cc52 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:40.925770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849f1db7-aec2-4b61-9d83-213a2644edbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:40.926823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f1db7-aec2-4b61-9d83-213a2644edbb None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:40.927359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f1db7-aec2-4b61-9d83-213a2644edbb None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:40.927834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f1db7-aec2-4b61-9d83-213a2644edbb None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:40.928727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849f1db7-aec2-4b61-9d83-213a2644edbb None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:40.929289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849f1db7-aec2-4b61-9d83-213a2644edbb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:40.930945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849f1db7-aec2-4b61-9d83-213a2644edbb None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:40.937190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:40.944740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 339/666] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:40 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:40.951651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:41.105143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f8bbe-307c-4200-9757-35aee9beef33 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:41.122493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:41.151182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b642691-a9a2-44c9-b508-34b71a356ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:41.152490 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b642691-a9a2-44c9-b508-34b71a356ce1 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:41.152768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b642691-a9a2-44c9-b508-34b71a356ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:41.153041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b642691-a9a2-44c9-b508-34b71a356ce1 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:41.153740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b642691-a9a2-44c9-b508-34b71a356ce1 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:41.154084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b642691-a9a2-44c9-b508-34b71a356ce1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:41.155693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b642691-a9a2-44c9-b508-34b71a356ce1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:41.161803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 328/667] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:40 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:42.098469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de73bcf-c4dd-4db0-a467-7428d80d1cbc None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:42.099047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de73bcf-c4dd-4db0-a467-7428d80d1cbc None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:42.099047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de73bcf-c4dd-4db0-a467-7428d80d1cbc None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:42.100774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 340/668] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:42.107107 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15331364-ee8e-4c2b-961f-4853ff62f188 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:42.107661 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15331364-ee8e-4c2b-961f-4853ff62f188 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:42.107661 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15331364-ee8e-4c2b-961f-4853ff62f188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:42.127145 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-15331364-ee8e-4c2b-961f-4853ff62f188 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:42.128392 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-15331364-ee8e-4c2b-961f-4853ff62f188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:42.212215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 329/669] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:42 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:42.482311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de73bcf-c4dd-4db0-a467-7428d80d1cbc None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:42.612527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de73bcf-c4dd-4db0-a467-7428d80d1cbc None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:42.649661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf7f25-c4a4-40f2-8f84-9d000de3d85e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:42.650294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf7f25-c4a4-40f2-8f84-9d000de3d85e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:42.650670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf7f25-c4a4-40f2-8f84-9d000de3d85e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:42.650947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf7f25-c4a4-40f2-8f84-9d000de3d85e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:42.662413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdf7f25-c4a4-40f2-8f84-9d000de3d85e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-9fdf7f25-c4a4-40f2-8f84-9d000de3d85e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:38:42.667953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:42.668920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:42.669782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 341/670] 10.222.0.39 () {56 vars in 1120 bytes} [Mon Oct 24 08:38:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:42.675161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15331364-ee8e-4c2b-961f-4853ff62f188 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:42.782066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15331364-ee8e-4c2b-961f-4853ff62f188 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:42.784527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:42.819316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:42.819918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:42.820221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:42.820490 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:42.821140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:42.821440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:42.823896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:42.832556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 330/671] 10.222.0.39 () {56 vars in 1115 bytes} [Mon Oct 24 08:38:42 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:42.838503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf7f25-c4a4-40f2-8f84-9d000de3d85e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:42.905066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:42.961209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf7f25-c4a4-40f2-8f84-9d000de3d85e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:43.007054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61742378-762c-4c87-a51c-a061e66e915e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:43.007689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61742378-762c-4c87-a51c-a061e66e915e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.007971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61742378-762c-4c87-a51c-a061e66e915e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.008278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61742378-762c-4c87-a51c-a061e66e915e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.021526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61742378-762c-4c87-a51c-a061e66e915e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:43.022987 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61742378-762c-4c87-a51c-a061e66e915e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-61742378-762c-4c87-a51c-a061e66e915e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:43.039289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:43.040304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:43.041337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:43.041337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:43.041337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:43.044275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 342/672] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:43.046748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:43.059103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac765f7f-4868-40e5-9f44-8e5be69f7afc None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.059103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac765f7f-4868-40e5-9f44-8e5be69f7afc None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.059103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac765f7f-4868-40e5-9f44-8e5be69f7afc None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.059976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 343/673] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:43.065641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db0d79-45c7-4140-8557-b1e8a6cbc742 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.065929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db0d79-45c7-4140-8557-b1e8a6cbc742 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.066192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db0d79-45c7-4140-8557-b1e8a6cbc742 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.086945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-43db0d79-45c7-4140-8557-b1e8a6cbc742 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:43.088857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-43db0d79-45c7-4140-8557-b1e8a6cbc742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:43.164680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 344/674] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:43 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:43.167287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2233b24c-bbdf-4882-9427-67147e852d56 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.167768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2233b24c-bbdf-4882-9427-67147e852d56 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.168245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2233b24c-bbdf-4882-9427-67147e852d56 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.169809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 345/675] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:43.171794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff1f39f-06c9-4184-a6fa-36b5396fa889 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:43.175465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34785c8-f353-45f1-9c8e-adfe36acba9f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.176014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34785c8-f353-45f1-9c8e-adfe36acba9f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.176518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34785c8-f353-45f1-9c8e-adfe36acba9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.196416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f34785c8-f353-45f1-9c8e-adfe36acba9f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:43.198603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f34785c8-f353-45f1-9c8e-adfe36acba9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:43.209766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:43.210635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.211181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.211596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.212401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:43.213382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:43.215032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:43.233997 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 331/676] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:43.235470 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:43.240833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:43.266398 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:43.285813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 346/677] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:43 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:43.358101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e408dcd-9a2f-4e17-8baa-895e9bb4c954 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:43.396061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:43.397088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.397574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.398207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.406518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:43.408319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:43.422540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:43.423556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 24 08:38:43.424600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:43.431676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 332/678] 10.222.0.39 () {56 vars in 1117 bytes} [Mon Oct 24 08:38:43 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:43.433899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34785c8-f353-45f1-9c8e-adfe36acba9f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:43.469214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:43.557729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34785c8-f353-45f1-9c8e-adfe36acba9f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:43.570990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:43.590900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d229b3c-1b68-406f-b329-cc007746a734 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:43.595810 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:43.596449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.596733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.597007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.597732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:43.598044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:43.599458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:43.617934 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 347/679] 10.222.0.39 () {56 vars in 1149 bytes} [Mon Oct 24 08:38:43 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:43.618749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:43.628772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4befa536-c452-4cc0-9809-dcd73ae2e4d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:43.629383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4befa536-c452-4cc0-9809-dcd73ae2e4d6 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.629666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4befa536-c452-4cc0-9809-dcd73ae2e4d6 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.629936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4befa536-c452-4cc0-9809-dcd73ae2e4d6 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.630588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4befa536-c452-4cc0-9809-dcd73ae2e4d6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:43.631948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4befa536-c452-4cc0-9809-dcd73ae2e4d6 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:43.647645 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 333/680] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:38:43 2022] POST /identity/v3/services => generated 241 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:38:43.648899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe152f-b2c3-42ef-b279-70e445e14e35 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.649216 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe152f-b2c3-42ef-b279-70e445e14e35 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.649516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe152f-b2c3-42ef-b279-70e445e14e35 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.650783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 334/681] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:43.652350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe152f-b2c3-42ef-b279-70e445e14e35 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:43.658033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:43.738879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da97665-9481-4534-855e-6beccb50d5d0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:43.775116 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe152f-b2c3-42ef-b279-70e445e14e35 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:43.780785 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e7f3c7-728a-4faa-ad6a-576a9ce15265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:43.781499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e7f3c7-728a-4faa-ad6a-576a9ce15265 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.781778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e7f3c7-728a-4faa-ad6a-576a9ce15265 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.782103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e7f3c7-728a-4faa-ad6a-576a9ce15265 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.782694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e7f3c7-728a-4faa-ad6a-576a9ce15265 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:43.784293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e7f3c7-728a-4faa-ad6a-576a9ce15265 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-a3e7f3c7-728a-4faa-ad6a-576a9ce15265 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:43.794177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:43.795228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:43.796661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 348/682] 10.222.0.39 () {56 vars in 1132 bytes} [Mon Oct 24 08:38:43 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:43.797494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a949358e-de7a-412b-818c-743dbc956cda None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.797494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a949358e-de7a-412b-818c-743dbc956cda None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.797494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a949358e-de7a-412b-818c-743dbc956cda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.816295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a949358e-de7a-412b-818c-743dbc956cda None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:43.817522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a949358e-de7a-412b-818c-743dbc956cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:43.826878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f97916-e8a9-4437-b315-2b4bb0dc6595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:43.827916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f97916-e8a9-4437-b315-2b4bb0dc6595 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:43.827916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f97916-e8a9-4437-b315-2b4bb0dc6595 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:43.828079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f97916-e8a9-4437-b315-2b4bb0dc6595 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:43.845039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:43.911417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f97916-e8a9-4437-b315-2b4bb0dc6595 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:43.912852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f97916-e8a9-4437-b315-2b4bb0dc6595 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:43.919619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 349/683] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:43 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:43.920507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a949358e-de7a-412b-818c-743dbc956cda None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:43.960822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:44.040101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 335/684] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:38:43 2022] GET /identity/v3/auth/tokens => generated 1373 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 24 08:38:44.047471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f97916-e8a9-4437-b315-2b4bb0dc6595 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:44.072761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a949358e-de7a-412b-818c-743dbc956cda None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:44.108781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:44.121185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:44.122346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:44.122802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:44.123212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:44.124050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:44.124558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:44.126065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:44.138798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 350/685] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:43 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:44.140649 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:44.186109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f97916-e8a9-4437-b315-2b4bb0dc6595 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:44.232723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:44.233458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:44.233946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:44.234217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:44.247174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:44.248666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:44.261670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:38:44.265005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:44.265968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:38:44.266883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:38:44.266883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:38:44.266883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:44.266883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 336/686] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:44.266883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:44.283490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa94218-962c-4e90-8695-16bf05e1e907 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:44.323534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:44.324159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:44.324486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:44.324707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:44.325584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:44.325707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:44.326964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:44.337668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 351/687] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:44 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:44.337668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:44.383612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:44.392228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d9b376-29a9-42de-b8ed-36bfc42b869e None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:44.440290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a78560-ae93-42bd-9b64-f561104123a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:44.441320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a78560-ae93-42bd-9b64-f561104123a0 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:44.441807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a78560-ae93-42bd-9b64-f561104123a0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:44.442331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a78560-ae93-42bd-9b64-f561104123a0 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:44.443235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a78560-ae93-42bd-9b64-f561104123a0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:44.444904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a78560-ae93-42bd-9b64-f561104123a0 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:44.461608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a407a2-e2e6-4bd8-b97a-6387e299979e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:44.469821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 337/688] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:38:44 2022] POST /identity/v3/services => generated 251 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:44.471859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a78560-ae93-42bd-9b64-f561104123a0 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:44.522249 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:44.523051 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:44.523759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:44.524169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:44.526276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:44.526711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:44.528328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:44.544186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 352/689] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:38:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:44.551741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:44.569392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:44.634022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a78560-ae93-42bd-9b64-f561104123a0 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:44.691290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791ee1b3-e3d7-412b-ab22-40eccf32230b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:44.691832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ee1b3-e3d7-412b-ab22-40eccf32230b None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:44.692111 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ee1b3-e3d7-412b-ab22-40eccf32230b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:44.692444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ee1b3-e3d7-412b-ab22-40eccf32230b None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:44.693116 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791ee1b3-e3d7-412b-ab22-40eccf32230b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:44.694485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791ee1b3-e3d7-412b-ab22-40eccf32230b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:44.702949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1935f3f3-f5bb-4602-bb64-a8828c14970d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:44.721192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 338/690] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:38:44 2022] POST /identity/v3/endpoints => generated 347 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:44.765755 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:44.766015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:44.766200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:44.766415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:44.775220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:44.776891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:44.790948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:44.793581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:44.794542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 353/691] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:44.799083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791ee1b3-e3d7-412b-ab22-40eccf32230b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:44.813898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:44.921585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791ee1b3-e3d7-412b-ab22-40eccf32230b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:44.961516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e9f757-9dc1-4c89-914e-0d89addca2a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:44.962341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9f757-9dc1-4c89-914e-0d89addca2a8 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:44.962787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9f757-9dc1-4c89-914e-0d89addca2a8 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:44.964557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9f757-9dc1-4c89-914e-0d89addca2a8 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:44.965408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e9f757-9dc1-4c89-914e-0d89addca2a8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:44.965859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e9f757-9dc1-4c89-914e-0d89addca2a8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:44.967438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e9f757-9dc1-4c89-914e-0d89addca2a8 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:44.982782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 339/692] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:38:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:44.996159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:45.063443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:45.107953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d425eb4-b506-457e-bef5-75094b63350e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:45.143297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11d7cd0-f7c0-46e2-97c8-13e6c3c7e8c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:45.143899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d7cd0-f7c0-46e2-97c8-13e6c3c7e8c0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:45.144205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d7cd0-f7c0-46e2-97c8-13e6c3c7e8c0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:45.144484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d7cd0-f7c0-46e2-97c8-13e6c3c7e8c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:45.145212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11d7cd0-f7c0-46e2-97c8-13e6c3c7e8c0 None admin] RBAC: Adding query filter params (role.id=eb1426874ce540f2a0d2c7f30c3b87f6, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=128903595c5e454c8c51c371d014eb49) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:45.145525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11d7cd0-f7c0-46e2-97c8-13e6c3c7e8c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:45.147002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11d7cd0-f7c0-46e2-97c8-13e6c3c7e8c0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:45.154098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 354/693] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:38:44 2022] GET /identity/v3/role_assignments?user.id=128903595c5e454c8c51c371d014eb49&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=eb1426874ce540f2a0d2c7f30c3b87f6 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:45.952636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d53fa-e89c-4568-adfe-e4b358e81084 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:45.953158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d53fa-e89c-4568-adfe-e4b358e81084 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:45.953158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d53fa-e89c-4568-adfe-e4b358e81084 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:45.954792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 340/694] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:45.959994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cf8f0a-888a-4855-b985-c0b6467beeae None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:45.960416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cf8f0a-888a-4855-b985-c0b6467beeae None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:45.960601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cf8f0a-888a-4855-b985-c0b6467beeae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:45.983061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a9cf8f0a-888a-4855-b985-c0b6467beeae None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:45.984265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a9cf8f0a-888a-4855-b985-c0b6467beeae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:46.058744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 355/695] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:46.328220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250d53fa-e89c-4568-adfe-e4b358e81084 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:46.404963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:46.439806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250d53fa-e89c-4568-adfe-e4b358e81084 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:46.478029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:46.478729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:46.479014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:46.479290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:46.479902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:46.481469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:46.491548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:46.492693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:46.493538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 341/696] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:46 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:46.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cf8f0a-888a-4855-b985-c0b6467beeae None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:46.570359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:46.623787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cf8f0a-888a-4855-b985-c0b6467beeae None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:46.666126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:46.666861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:46.667135 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:46.667406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:46.668043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] RBAC: Adding query filter params (name=object-store) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:46.668385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:46.669781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:46.675155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 356/697] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:46 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:46.681372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:46.785731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09f64c-2ab6-4a92-befd-218d57368099 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:46.828897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddca65d-94b3-4aad-bc8b-bbbc55660310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:46.829578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddca65d-94b3-4aad-bc8b-bbbc55660310 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:46.829858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddca65d-94b3-4aad-bc8b-bbbc55660310 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:46.830126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddca65d-94b3-4aad-bc8b-bbbc55660310 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:46.830772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddca65d-94b3-4aad-bc8b-bbbc55660310 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:46.831075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddca65d-94b3-4aad-bc8b-bbbc55660310 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:46.832593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddca65d-94b3-4aad-bc8b-bbbc55660310 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:46.834086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:46.841813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 342/698] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:46 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:46.847837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:46.954320 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84be7782-7a1d-4221-9a1e-b2dacf1199ec None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:46.957469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca26d3a1-ffc7-4d32-9648-fe6ef0c4582b None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:46.958336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca26d3a1-ffc7-4d32-9648-fe6ef0c4582b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:46.958813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca26d3a1-ffc7-4d32-9648-fe6ef0c4582b None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:46.960630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 343/699] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:46.966774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8732e6f-1d1c-4632-a527-00e1b60bf528 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:46.967321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8732e6f-1d1c-4632-a527-00e1b60bf528 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:46.967837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8732e6f-1d1c-4632-a527-00e1b60bf528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:46.980634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:46.998671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:46.999725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.000278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.000459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b8732e6f-1d1c-4632-a527-00e1b60bf528 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:47.000922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.001704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b8732e6f-1d1c-4632-a527-00e1b60bf528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:47.002120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] RBAC: Adding query filter params (interface=public, service_id=d35c3d4264d94889b62a2ee09f6341b7, region_id=RegionOne) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:47.002710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:47.004345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:47.018650 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 357/700] 10.222.0.39 () {56 vars in 1251 bytes} [Mon Oct 24 08:38:46 2022] GET /identity/v3/endpoints?service_id=d35c3d4264d94889b62a2ee09f6341b7&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:47.021126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:47.092190 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 344/701] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:46 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:47.110695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53ed11-e85d-4050-865e-ec22a90135c5 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.110982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53ed11-e85d-4050-865e-ec22a90135c5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.111264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53ed11-e85d-4050-865e-ec22a90135c5 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.117337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 345/702] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:47 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:47.118177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c7880-1a72-4799-8348-6345a6921c93 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.118466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c7880-1a72-4799-8348-6345a6921c93 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.118745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c7880-1a72-4799-8348-6345a6921c93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.139155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-396c7880-1a72-4799-8348-6345a6921c93 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:47.140405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-396c7880-1a72-4799-8348-6345a6921c93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:47.155029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c903a81e-363f-42d2-a08a-1b7089374223 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:47.192818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2baf84e-95fe-4c03-93d3-34ec0a37d1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:47.193438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2baf84e-95fe-4c03-93d3-34ec0a37d1dd None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.193716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2baf84e-95fe-4c03-93d3-34ec0a37d1dd None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.193997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2baf84e-95fe-4c03-93d3-34ec0a37d1dd None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.194604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2baf84e-95fe-4c03-93d3-34ec0a37d1dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:47.196000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2baf84e-95fe-4c03-93d3-34ec0a37d1dd None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:47.202931 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 358/703] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:38:47 2022] GET /identity/v3/services => generated 1497 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:38:47.222191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 346/704] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:47.384436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2baf84e-95fe-4c03-93d3-34ec0a37d1dd None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:47.448692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:47.505137 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396c7880-1a72-4799-8348-6345a6921c93 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:47.506166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2baf84e-95fe-4c03-93d3-34ec0a37d1dd None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:47.546549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97a0655-5805-4a5e-b86a-55093c05a613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:47.547608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97a0655-5805-4a5e-b86a-55093c05a613 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.548089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97a0655-5805-4a5e-b86a-55093c05a613 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.548590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97a0655-5805-4a5e-b86a-55093c05a613 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.549507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97a0655-5805-4a5e-b86a-55093c05a613 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:47.551355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97a0655-5805-4a5e-b86a-55093c05a613 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-e97a0655-5805-4a5e-b86a-55093c05a613 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:47.565480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:47.566882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:47.568766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 359/705] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:47 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:47.569613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada67590-8f2a-4009-89ea-f5199cd1640a None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.569613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada67590-8f2a-4009-89ea-f5199cd1640a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.569613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada67590-8f2a-4009-89ea-f5199cd1640a None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.570390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 360/706] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:47.576804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add0ae4-4baa-48fe-a3e3-bae3d0c95001 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.577784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add0ae4-4baa-48fe-a3e3-bae3d0c95001 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.577784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add0ae4-4baa-48fe-a3e3-bae3d0c95001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.585073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:47.603123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5add0ae4-4baa-48fe-a3e3-bae3d0c95001 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:47.604857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5add0ae4-4baa-48fe-a3e3-bae3d0c95001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:47.641921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396c7880-1a72-4799-8348-6345a6921c93 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:47.680313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:47.680968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.681249 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.681522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] PATH_INFO: `/v3/services/compute` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.682644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:47.684023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:47.687676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 361/707] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:47.693026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add0ae4-4baa-48fe-a3e3-bae3d0c95001 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:47.693389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:47.694376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:47.695223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:47.700435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 347/708] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:47 2022] GET /identity/v3/services/compute => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:47.701329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:47.759678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:47.818766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc236bd8-8d5d-4ad1-8270-4b4587559835 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:47.835478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add0ae4-4baa-48fe-a3e3-bae3d0c95001 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:47.842541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:47.856678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:47.857523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.858023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.858454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.859334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] RBAC: Adding query filter params (name=compute) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:47.859782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:47.861318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:47.867073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 348/709] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:47 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:47.872907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:47.882927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:47.883794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:47.884638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:47.885133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:47.886346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:47.887127 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:47.888807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:47.894760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 362/710] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:47 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:47.903147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:47.998357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69c051-2eba-460d-a96f-2db946bda03c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:48.027350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a09f0a9-57ca-4a22-99a8-559c3f05620c None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:48.035206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:48.036039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:48.036343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:48.036631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:48.037231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] RBAC: Adding query filter params (type=compute) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:48.037533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:48.038930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:48.039334 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:48.044263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 349/711] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:47 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:48.045077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:48.074421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:48.075534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:48.076024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:48.076675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:48.077833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:48.078886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:48.080994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:48.089025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 363/712] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:47 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:48.097951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:48.109760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:48.179114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fda2e4-8819-4d1a-bcc0-c69c69265cba None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:48.224576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cb38a4-5fb9-4cd1-a892-098b655f1874 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:48.237112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:48.238554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:48.239351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:48.239832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] PATH_INFO: `/v3/services/image` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:48.240668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:48.242614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:48.258402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:48.259447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:48.261295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:48.263354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 350/713] 10.222.0.39 () {56 vars in 1105 bytes} [Mon Oct 24 08:38:48 2022] GET /identity/v3/services/image => generated 86 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:48.269643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:48.270184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:48.270303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:48.270516 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:48.270788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:48.271529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] RBAC: Adding query filter params (interface=public, service_id=dfac4fb97a8346a1b3803f01ffd5d039, region_id=RegionOne) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:48.271899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:48.273318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:48.279259 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 364/714] 10.222.0.39 () {56 vars in 1251 bytes} [Mon Oct 24 08:38:48 2022] GET /identity/v3/endpoints?service_id=dfac4fb97a8346a1b3803f01ffd5d039&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:48.285618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:48.326431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:48.384537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82da3c6-a5a5-4683-841f-9860bd2084c4 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:48.403392 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:48.411196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5622ae37-b5d8-475f-9488-a0f427d6f150 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:48.436028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:48.437609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:48.438267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:48.438755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:48.440295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] RBAC: Adding query filter params (name=image) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:48.441070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:48.443284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:48.447945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e8fa4-6bda-4128-b770-47b173af127b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:48.448494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e8fa4-6bda-4128-b770-47b173af127b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:48.448762 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e8fa4-6bda-4128-b770-47b173af127b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:48.449056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e8fa4-6bda-4128-b770-47b173af127b None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:48.449631 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77e8fa4-6bda-4128-b770-47b173af127b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:48.451044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77e8fa4-6bda-4128-b770-47b173af127b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:48.452285 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 351/715] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:48 2022] GET /identity/v3/services?name=image => generated 122 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:48.464125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:48.464362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 365/716] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:38:48 2022] GET /identity/v3/services => generated 1497 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:38:48.592224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c400f9-b463-4c2a-9e46-71d344dbb71c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:48.633865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7631b663-1c28-40d9-b434-19a3083f7b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:48.635261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7631b663-1c28-40d9-b434-19a3083f7b45 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:48.635690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7631b663-1c28-40d9-b434-19a3083f7b45 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:48.636220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7631b663-1c28-40d9-b434-19a3083f7b45 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:48.637070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7631b663-1c28-40d9-b434-19a3083f7b45 None admin] RBAC: Adding query filter params (type=image) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:48.637547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7631b663-1c28-40d9-b434-19a3083f7b45 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:48.639118 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7631b663-1c28-40d9-b434-19a3083f7b45 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:48.644730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 352/717] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:48 2022] GET /identity/v3/services?type=image => generated 122 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:49.562201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b13e81-720b-4f4f-8935-48fa17c57696 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:49.562791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b13e81-720b-4f4f-8935-48fa17c57696 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:49.562791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b13e81-720b-4f4f-8935-48fa17c57696 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:49.564450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 366/718] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:49.570575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056dfe81-9f71-4b23-9183-456e44315855 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:49.572053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056dfe81-9f71-4b23-9183-456e44315855 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:49.572053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056dfe81-9f71-4b23-9183-456e44315855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:49.590697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-056dfe81-9f71-4b23-9183-456e44315855 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:49.591898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-056dfe81-9f71-4b23-9183-456e44315855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:49.631065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:49.668044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 353/719] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:49.940544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b13e81-720b-4f4f-8935-48fa17c57696 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:50.060526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:50.065373 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b13e81-720b-4f4f-8935-48fa17c57696 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:50.105205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:50.106083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.106540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.106954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.107662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:50.109199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:50.118786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:50.120364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:50.121222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:50.121992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 367/720] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:49 2022] GET /identity/v3/services/object-store => generated 93 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:50.127726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056dfe81-9f71-4b23-9183-456e44315855 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:50.248947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056dfe81-9f71-4b23-9183-456e44315855 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:50.292779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59269d84-e9f5-457e-a084-4dc28ccd28ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:50.293480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59269d84-e9f5-457e-a084-4dc28ccd28ac None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.293770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59269d84-e9f5-457e-a084-4dc28ccd28ac None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.294097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59269d84-e9f5-457e-a084-4dc28ccd28ac None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.294824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59269d84-e9f5-457e-a084-4dc28ccd28ac None admin] RBAC: Adding query filter params (name=object-store) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:50.295158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59269d84-e9f5-457e-a084-4dc28ccd28ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:50.296786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59269d84-e9f5-457e-a084-4dc28ccd28ac None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:50.303583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 354/721] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:50 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:50.310410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:50.432751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f9c3d-b3e7-4269-b9b8-634aee081628 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.433111 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f9c3d-b3e7-4269-b9b8-634aee081628 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.433397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f9c3d-b3e7-4269-b9b8-634aee081628 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.435010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 355/722] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:50.440407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fccb90-8bcf-4252-81d8-33c5454c5ad2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.440701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fccb90-8bcf-4252-81d8-33c5454c5ad2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.440976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fccb90-8bcf-4252-81d8-33c5454c5ad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.461337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-d0fccb90-8bcf-4252-81d8-33c5454c5ad2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:50.462533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d0fccb90-8bcf-4252-81d8-33c5454c5ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:50.472168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b552a7-0cc5-4f28-adeb-561e6dcc4f96 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:50.495967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:50.522891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:50.524400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.526459 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.527285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.529244 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:50.530225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:50.534264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:50.548355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 368/723] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:50 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:50.548833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 356/724] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:50.550013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:50.583940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:50.671594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab5195-e16a-403c-907c-4ad23048db29 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:50.714337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1771269-4936-434a-b26b-b635e65a551f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:50.715361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1771269-4936-434a-b26b-b635e65a551f None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.715835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1771269-4936-434a-b26b-b635e65a551f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.716327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1771269-4936-434a-b26b-b635e65a551f None admin] PATH_INFO: `/v3/endpoints` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.717295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1771269-4936-434a-b26b-b635e65a551f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:50.718973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1771269-4936-434a-b26b-b635e65a551f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:50.748022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 369/725] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:38:50 2022] POST /identity/v3/endpoints => generated 349 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:50.820744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fccb90-8bcf-4252-81d8-33c5454c5ad2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:50.838248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b900d-ef1e-4055-ab0d-c17a33743fc2 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.838529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b900d-ef1e-4055-ab0d-c17a33743fc2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.838882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b900d-ef1e-4055-ab0d-c17a33743fc2 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.840721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 370/726] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:50.846499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7742eb-94a9-460b-af2b-679656b9fc68 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.846780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7742eb-94a9-460b-af2b-679656b9fc68 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.847057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7742eb-94a9-460b-af2b-679656b9fc68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.875863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6e7742eb-94a9-460b-af2b-679656b9fc68 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:50.878426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6e7742eb-94a9-460b-af2b-679656b9fc68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:50.951366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fccb90-8bcf-4252-81d8-33c5454c5ad2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:50.974012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 371/727] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:50.975469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77853d-835f-4721-b06f-ca9c9bd5eec1 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.975760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77853d-835f-4721-b06f-ca9c9bd5eec1 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.976035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77853d-835f-4721-b06f-ca9c9bd5eec1 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:50.981683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 372/728] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:50.983615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a276088-de51-41c2-901e-330bd8ebc85f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:50.983906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a276088-de51-41c2-901e-330bd8ebc85f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:50.984238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a276088-de51-41c2-901e-330bd8ebc85f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:51.005057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0a276088-de51-41c2-901e-330bd8ebc85f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:51.005630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0a276088-de51-41c2-901e-330bd8ebc85f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:51.007277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fc6ed5-d681-4c4a-ac30-ce33e486cef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:51.007889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc6ed5-d681-4c4a-ac30-ce33e486cef9 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:51.010345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc6ed5-d681-4c4a-ac30-ce33e486cef9 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:51.010345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc6ed5-d681-4c4a-ac30-ce33e486cef9 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:51.010345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fc6ed5-d681-4c4a-ac30-ce33e486cef9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:51.011036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fc6ed5-d681-4c4a-ac30-ce33e486cef9 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:51.011669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:51.032158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 357/729] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:38:50 2022] POST /identity/v3/services => generated 242 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:51.104570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 373/730] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:50 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:51.260970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fc6ed5-d681-4c4a-ac30-ce33e486cef9 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:51.291913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:51.370487 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fc6ed5-d681-4c4a-ac30-ce33e486cef9 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:51.407492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:51.408339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:51.408718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:51.409087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:51.409772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:51.411320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:51.412521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a276088-de51-41c2-901e-330bd8ebc85f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:51.423248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:51.424573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 24 08:38:51.425625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:51.426754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 358/731] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:51 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:51.433387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:51.538809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a276088-de51-41c2-901e-330bd8ebc85f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:51.561107 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:51.575486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71ba7f4-7c08-48df-a9bc-e1c64ffc2af5 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:51.582901 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbe26ce-87c5-4ca5-b234-497429cf80f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:51.583517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbe26ce-87c5-4ca5-b234-497429cf80f9 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:51.583797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbe26ce-87c5-4ca5-b234-497429cf80f9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:51.584071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbe26ce-87c5-4ca5-b234-497429cf80f9 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:51.584937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbe26ce-87c5-4ca5-b234-497429cf80f9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:51.586809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbe26ce-87c5-4ca5-b234-497429cf80f9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:51.592999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:51.608722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 374/732] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:38:51 2022] POST /identity/v3/services => generated 242 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:51.636476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:51.637443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:51.637896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:51.638325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:51.639200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:51.639671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:51.641293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:51.648165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 359/733] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:51 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:51.654762 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbe26ce-87c5-4ca5-b234-497429cf80f9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:51.782805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbe26ce-87c5-4ca5-b234-497429cf80f9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:51.822027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b00da6-cbd4-47de-bbd1-c6161a2a13db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:51.823989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b00da6-cbd4-47de-bbd1-c6161a2a13db None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:51.824333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b00da6-cbd4-47de-bbd1-c6161a2a13db None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:51.824567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b00da6-cbd4-47de-bbd1-c6161a2a13db None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:51.825192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b00da6-cbd4-47de-bbd1-c6161a2a13db None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:51.825495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b00da6-cbd4-47de-bbd1-c6161a2a13db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:51.826845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b00da6-cbd4-47de-bbd1-c6161a2a13db None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:51.833094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 375/734] 10.222.0.39 () {56 vars in 1114 bytes} [Mon Oct 24 08:38:51 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:51.840004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:51.949592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae198204-eb08-48da-8d81-9f0fd0a18658 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:51.986660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc68977-c992-4c0f-bc24-de5e7e90ccf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:51.987523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc68977-c992-4c0f-bc24-de5e7e90ccf0 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:51.987981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc68977-c992-4c0f-bc24-de5e7e90ccf0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:51.988423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc68977-c992-4c0f-bc24-de5e7e90ccf0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:51.989252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc68977-c992-4c0f-bc24-de5e7e90ccf0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:51.996153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc68977-c992-4c0f-bc24-de5e7e90ccf0 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:52.014083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 360/735] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:38:51 2022] POST /identity/v3/endpoints => generated 346 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:53.118849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622cf269-a1a5-4512-867e-a704a8c110c3 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:53.119700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622cf269-a1a5-4512-867e-a704a8c110c3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:53.119700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622cf269-a1a5-4512-867e-a704a8c110c3 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:53.121414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 376/736] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:53.126700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71ac2f-dce3-4026-9af6-2cb217449eb4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:53.127129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71ac2f-dce3-4026-9af6-2cb217449eb4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:53.127278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71ac2f-dce3-4026-9af6-2cb217449eb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:53.141334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:53.148099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3d71ac2f-dce3-4026-9af6-2cb217449eb4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:53.149849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3d71ac2f-dce3-4026-9af6-2cb217449eb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:53.229756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 361/737] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:53 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:53.350427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114683ac-a51b-4241-a992-12b2c28bc5e3 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:53.350623 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114683ac-a51b-4241-a992-12b2c28bc5e3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:53.350879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114683ac-a51b-4241-a992-12b2c28bc5e3 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:53.352503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 377/738] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:53.358500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d025a0-2a0a-45f8-a419-59c05d5fcdb3 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:53.358789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d025a0-2a0a-45f8-a419-59c05d5fcdb3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:53.359065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d025a0-2a0a-45f8-a419-59c05d5fcdb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:53.379226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f7d025a0-2a0a-45f8-a419-59c05d5fcdb3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:53.380638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f7d025a0-2a0a-45f8-a419-59c05d5fcdb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:53.456952 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 362/739] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:53 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:53.511578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114683ac-a51b-4241-a992-12b2c28bc5e3 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:53.591153 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:53.648368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114683ac-a51b-4241-a992-12b2c28bc5e3 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:53.684532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:53.685167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:53.685443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:53.685714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:53.686297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:53.687640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:53.697309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:53.698546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 378/740] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:53 2022] GET /identity/v3/services/object-store => generated 93 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:53.704560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d025a0-2a0a-45f8-a419-59c05d5fcdb3 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:53.771657 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:53.774162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:53.814147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d025a0-2a0a-45f8-a419-59c05d5fcdb3 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:53.854222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:53.855208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:53.855726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:53.856237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:53.857343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:53.857868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:53.859482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:53.865237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 363/741] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:53 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:53.872211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:53.896336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665a0573-aafe-4522-8871-a1b8cd62206e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:53.923747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:53.937557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:53.938391 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:53.938666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:53.938935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] PATH_INFO: `/v3/services/compute` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:53.939831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:53.941228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:53.954002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:53.955364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:53.956584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:53.960226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 379/742] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:53 2022] GET /identity/v3/services/compute => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:53.961437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:54.025052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6cbd60-a03b-4bcf-9fc3-d1e591f9c1c2 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:54.067107 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:54.082214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:54.083232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.084491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.084810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.085418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:54.085723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:54.087130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:54.098584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 364/743] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:53 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:54.099577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:54.120705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7af078-13dc-4fb8-8561-2d12f3c98c40 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:54.165267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8dd381-3b79-4c29-b6e6-a33cfbae1d32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:54.167084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8dd381-3b79-4c29-b6e6-a33cfbae1d32 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.167614 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8dd381-3b79-4c29-b6e6-a33cfbae1d32 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.168102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8dd381-3b79-4c29-b6e6-a33cfbae1d32 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.169232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8dd381-3b79-4c29-b6e6-a33cfbae1d32 None admin] RBAC: Adding query filter params (name=compute) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:54.170185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8dd381-3b79-4c29-b6e6-a33cfbae1d32 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:54.172041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8dd381-3b79-4c29-b6e6-a33cfbae1d32 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:54.178009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 380/744] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:53 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:54.179615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dabf4b-e4b8-4e96-9956-7410b4dc5beb None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.180205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dabf4b-e4b8-4e96-9956-7410b4dc5beb None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.180681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dabf4b-e4b8-4e96-9956-7410b4dc5beb None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.182710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 381/745] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:54.186128 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dabf4b-e4b8-4e96-9956-7410b4dc5beb None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:54.227514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6815731-244f-4f44-b3cb-5ef6d262bcdb None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:54.258796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:54.273043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6c326b-260b-4b7a-a6e2-6fc0bf68ffe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:54.273653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6c326b-260b-4b7a-a6e2-6fc0bf68ffe6 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.273927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6c326b-260b-4b7a-a6e2-6fc0bf68ffe6 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.274189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6c326b-260b-4b7a-a6e2-6fc0bf68ffe6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.274956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6c326b-260b-4b7a-a6e2-6fc0bf68ffe6 None admin] RBAC: Adding query filter params (interface=admin, service_id=d35c3d4264d94889b62a2ee09f6341b7, region_id=RegionOne) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:54.275262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6c326b-260b-4b7a-a6e2-6fc0bf68ffe6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:54.277780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6c326b-260b-4b7a-a6e2-6fc0bf68ffe6 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:54.283154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 365/746] 10.222.0.39 () {56 vars in 1249 bytes} [Mon Oct 24 08:38:54 2022] GET /identity/v3/endpoints?service_id=d35c3d4264d94889b62a2ee09f6341b7&interface=admin®ion_id=RegionOne => generated 193 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:54.284396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84008264-5d11-4df1-9bef-30f11017af38 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.284666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84008264-5d11-4df1-9bef-30f11017af38 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.284933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84008264-5d11-4df1-9bef-30f11017af38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.309796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-84008264-5d11-4df1-9bef-30f11017af38 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:54.310019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-84008264-5d11-4df1-9bef-30f11017af38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:54.329182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dabf4b-e4b8-4e96-9956-7410b4dc5beb None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:54.373740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:54.374350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.374654 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.374916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.375051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:54.375549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] RBAC: Adding query filter params (type=compute) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:54.375851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:54.377284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:54.382470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 382/747] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:54 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:54.383475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:54.396178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 366/748] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:54 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:54.397441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69430cb-12d3-4f42-997c-b085f8dcab06 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.397725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69430cb-12d3-4f42-997c-b085f8dcab06 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.398002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69430cb-12d3-4f42-997c-b085f8dcab06 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.399396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 367/749] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:54.399971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69430cb-12d3-4f42-997c-b085f8dcab06 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:54.497057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc3a332-20ff-4d49-a480-531996663176 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:54.517099 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:54.540817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69430cb-12d3-4f42-997c-b085f8dcab06 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:54.548962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b8129f-c3c6-41fb-a8c0-abe2445fbc35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:54.549899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b8129f-c3c6-41fb-a8c0-abe2445fbc35 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.550346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b8129f-c3c6-41fb-a8c0-abe2445fbc35 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.550790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b8129f-c3c6-41fb-a8c0-abe2445fbc35 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.551776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b8129f-c3c6-41fb-a8c0-abe2445fbc35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:54.554150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b8129f-c3c6-41fb-a8c0-abe2445fbc35 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:54.572944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 383/750] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:38:54 2022] GET /identity/v3/services => generated 1957 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:38:54.572944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad76413-ddae-46c0-88c0-0b6d70e9469b None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.572944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad76413-ddae-46c0-88c0-0b6d70e9469b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.572944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad76413-ddae-46c0-88c0-0b6d70e9469b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.586957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0ad76413-ddae-46c0-88c0-0b6d70e9469b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:54.588361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0ad76413-ddae-46c0-88c0-0b6d70e9469b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:54.608339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:54.609493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.609839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.610446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.611972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] RBAC: Adding query filter params (interface=public, service_id=a76f9539e5a545ca9cdb415f30dbb835, region_id=RegionOne) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:54.612310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:54.613730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:54.622414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 368/751] 10.222.0.39 () {56 vars in 1251 bytes} [Mon Oct 24 08:38:54 2022] GET /identity/v3/endpoints?service_id=a76f9539e5a545ca9cdb415f30dbb835&interface=public®ion_id=RegionOne => generated 194 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:54.629267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:54.699857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 384/752] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:54 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:54.700971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad76413-ddae-46c0-88c0-0b6d70e9469b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:54.708277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:54.783961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d176dd-9fe8-460c-a2c7-5ae02c4878dd None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:54.840093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ec007f-69df-44b9-b5c4-a34ac2e4c0a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:54.840961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ec007f-69df-44b9-b5c4-a34ac2e4c0a3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.841302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ec007f-69df-44b9-b5c4-a34ac2e4c0a3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.841669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ec007f-69df-44b9-b5c4-a34ac2e4c0a3 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.842345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ec007f-69df-44b9-b5c4-a34ac2e4c0a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:54.844280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ec007f-69df-44b9-b5c4-a34ac2e4c0a3 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:54.851225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad76413-ddae-46c0-88c0-0b6d70e9469b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:54.853327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 369/753] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:38:54 2022] GET /identity/v3/services => generated 1957 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:38:54.865413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:54.896803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:54.897521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:54.897798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:54.898109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] PATH_INFO: `/v3/services/image` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:54.898792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:54.900208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:54.911116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:54.912677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:54.914589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 385/754] 10.222.0.39 () {56 vars in 1105 bytes} [Mon Oct 24 08:38:54 2022] GET /identity/v3/services/image => generated 86 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:54.919125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ec007f-69df-44b9-b5c4-a34ac2e4c0a3 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:54.985224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:55.070983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ec007f-69df-44b9-b5c4-a34ac2e4c0a3 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:55.090749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:55.135845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:55.136103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:55.136381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:55.136612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:55.137256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] RBAC: Adding query filter params (name=image) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:55.137559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:55.139706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:55.151720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 370/755] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:54 2022] GET /identity/v3/services?name=image => generated 122 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:55.152948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228f01d0-e614-41ca-8aab-080cf666ca35 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:55.156831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:55.208392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:55.210406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:55.210946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:55.211432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:55.212349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:55.215233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:55.219096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING keystone.server.flask.common [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 24 08:38:55.246559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.sql.core [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 24 08:38:55.246559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 24 08:38:55.246559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [parameters: {'id': '68acf170f7d7462dbb536ebb06d6c570', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 24 08:38:55.246559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=93080) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 68acf170f7d7462dbb536ebb06d6c570.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 68acf170f7d7462dbb536ebb06d6c570. Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:38:55.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application result.read() Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 24 08:38:55.272706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application yield resource Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self.session.rollback() Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self.force_reraise() Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise self.value Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application session.commit() Oct 24 08:38:55.274466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._prepare_impl() Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self.session.flush() Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._flush(objects) Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application compat.raise_( Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application flush_context.execute() Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 24 08:38:55.278015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rec.execute(self) Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 24 08:38:55.281251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application result.read() Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:38:55.283461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [parameters: {'id': '68acf170f7d7462dbb536ebb06d6c570', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:55.289270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:55.295797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 68acf170f7d7462dbb536ebb06d6c570. Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 386/756] 10.222.0.39 () {60 vars in 1140 bytes} [Mon Oct 24 08:38:54 2022] POST /identity/v3/projects => generated 262 bytes in 288 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:55.300861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:55.328357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:55.343051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f98ee44-5df9-4b45-aa4c-a00bf872d658 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:55.392096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:55.393170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:55.393766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:55.394262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:55.395099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] RBAC: Adding query filter params (type=image) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:55.396280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:55.398447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:55.407721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 371/757] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:55 2022] GET /identity/v3/services?type=image => generated 350 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:55.413260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:55.428451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c03411-0a68-42d5-824e-b9989d3f276e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:55.467553 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:55.468228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:55.468526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:55.468824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:55.473595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:55.479993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:55.481993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:38:55.496098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:55.497212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:55.498119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:38:55.498119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:38:55.498119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:38:55.498119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:38:55.498119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:38:55.498119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:38:55.498119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:38:55.498119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:55.500913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 387/758] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:55.509000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:55.540930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3011cf-3962-4cbd-b48f-77ccadc655b2 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:55.584418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:55.585388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:55.585901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:55.586385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:55.587540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] RBAC: Adding query filter params (interface=public, service_id=24769591d8af42829a211ce6d4646e85, region_id=RegionOne) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:55.588803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:55.591012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:55.602028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 372/759] 10.222.0.39 () {56 vars in 1251 bytes} [Mon Oct 24 08:38:55 2022] GET /identity/v3/endpoints?service_id=24769591d8af42829a211ce6d4646e85&interface=public®ion_id=RegionOne => generated 194 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:55.608122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:55.623262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:55.636090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb6d15c-3d22-40f1-a5ab-44e7025e0f78 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:55.678944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec90242f-759f-4598-bb8c-094c6848a395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:55.679958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90242f-759f-4598-bb8c-094c6848a395 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:55.680512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90242f-759f-4598-bb8c-094c6848a395 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:55.680975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90242f-759f-4598-bb8c-094c6848a395 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:55.683032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90242f-759f-4598-bb8c-094c6848a395 None admin] RBAC: Adding query filter params (name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:55.683544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90242f-759f-4598-bb8c-094c6848a395 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:55.686751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90242f-759f-4598-bb8c-094c6848a395 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:55.736170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 388/760] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:55 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:55.749412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbcce24-e233-46a1-8b46-7bf2c394cdba None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:55.752858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:55.794241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f2876-15c4-479d-888e-bea3b4a77965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:55.795319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f2876-15c4-479d-888e-bea3b4a77965 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:55.795871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f2876-15c4-479d-888e-bea3b4a77965 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:55.796378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f2876-15c4-479d-888e-bea3b4a77965 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:55.797239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197f2876-15c4-479d-888e-bea3b4a77965 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:55.798876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197f2876-15c4-479d-888e-bea3b4a77965 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:55.807446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 373/761] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:38:55 2022] GET /identity/v3/services => generated 1957 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:38:56.898875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653ddf58-2087-422a-bc71-da98c22eac35 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:56.899411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653ddf58-2087-422a-bc71-da98c22eac35 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:56.899411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653ddf58-2087-422a-bc71-da98c22eac35 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:56.901219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 389/762] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:56.906924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982be1c-68ef-4149-a7b2-ba5e5f71fff1 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:56.907403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982be1c-68ef-4149-a7b2-ba5e5f71fff1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:56.907483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982be1c-68ef-4149-a7b2-ba5e5f71fff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:56.927749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5982be1c-68ef-4149-a7b2-ba5e5f71fff1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:56.928987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5982be1c-68ef-4149-a7b2-ba5e5f71fff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:57.014497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 374/763] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:56 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:57.225673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f2be9-94e6-46d6-b7e8-c8644c710c08 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:57.225921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f2be9-94e6-46d6-b7e8-c8644c710c08 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:57.226316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f2be9-94e6-46d6-b7e8-c8644c710c08 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:57.227779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 390/764] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:57.234945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa903f9b-0dc4-40cc-9e31-114ec1c6633e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:57.235452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa903f9b-0dc4-40cc-9e31-114ec1c6633e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:57.235943 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa903f9b-0dc4-40cc-9e31-114ec1c6633e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:57.266024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-fa903f9b-0dc4-40cc-9e31-114ec1c6633e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:57.268059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-fa903f9b-0dc4-40cc-9e31-114ec1c6633e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:57.326999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999f2be9-94e6-46d6-b7e8-c8644c710c08 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:57.385332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 375/765] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:57 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:57.465360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999f2be9-94e6-46d6-b7e8-c8644c710c08 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:57.511103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:57.511673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:57.511966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:57.512277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:57.512944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:57.514343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:57.524263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:57.526493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:57.528211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 391/766] 10.222.0.39 () {56 vars in 1126 bytes} [Mon Oct 24 08:38:57 2022] GET /identity/v3/services/object-store => generated 93 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:57.540718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa903f9b-0dc4-40cc-9e31-114ec1c6633e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:57.592197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:57.680810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa903f9b-0dc4-40cc-9e31-114ec1c6633e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:57.720249 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:57.720935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:57.721212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:57.721484 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:57.722168 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:57.722473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:57.723941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:57.731753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 376/767] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:57 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:57.738778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:57.759927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:57.766626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:57.883682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45450f-8dbb-4bb0-a9e7-36e173998036 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:57.896689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:57.911864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbb6a3c-606e-4f4f-ac01-29f5286c6333 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:57.945185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:57.948765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:57.949599 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:57.950470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:57.951348 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:57.952462 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:57.954545 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:57.964306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:57.965355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:57.965865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:57.966342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] PATH_INFO: `/v3/services/compute` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:57.967195 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:57.969144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:57.969686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 392/768] 10.222.0.39 () {56 vars in 1122 bytes} [Mon Oct 24 08:38:57 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:57.971054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:57.987355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:57.989231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 24 08:38:57.990903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:57.995023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 377/769] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:38:57 2022] GET /identity/v3/services/compute => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:58.002237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:58.097477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:58.131745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef9fb6b-9890-4e42-b876-674159f3337f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:58.166042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56897a6e-9aed-4c41-949a-ccfd65e25134 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:58.190486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2128c5-9c17-4a38-860d-9d6b78cb5302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:58.191497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2128c5-9c17-4a38-860d-9d6b78cb5302 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:58.191994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2128c5-9c17-4a38-860d-9d6b78cb5302 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:58.192509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2128c5-9c17-4a38-860d-9d6b78cb5302 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:58.193395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2128c5-9c17-4a38-860d-9d6b78cb5302 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:58.194970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2128c5-9c17-4a38-860d-9d6b78cb5302 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:58.222924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 393/770] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:38:57 2022] POST /identity/v3/endpoints => generated 325 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:58.225602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:58.226241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:58.226522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:58.226794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:58.227453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] RBAC: Adding query filter params (name=compute) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:58.227771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:58.229880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:58.236320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 378/771] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:58 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:38:58.246146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2128c5-9c17-4a38-860d-9d6b78cb5302 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:58.391338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2128c5-9c17-4a38-860d-9d6b78cb5302 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:58.442570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e42e1c-fe85-470e-a347-45d85fb26d01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:58.443491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e42e1c-fe85-470e-a347-45d85fb26d01 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:58.444058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e42e1c-fe85-470e-a347-45d85fb26d01 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:58.444541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e42e1c-fe85-470e-a347-45d85fb26d01 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:58.445188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e42e1c-fe85-470e-a347-45d85fb26d01 None admin] RBAC: Adding query filter params (type=compute) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:58.445492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e42e1c-fe85-470e-a347-45d85fb26d01 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:58.447041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e42e1c-fe85-470e-a347-45d85fb26d01 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:58.458823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 394/772] 10.222.0.39 () {56 vars in 1112 bytes} [Mon Oct 24 08:38:58 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:58.460304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:58.482468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:58.546846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5155f5fe-8c4d-4ec7-9572-3245c9a37dcf None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:58.547360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5155f5fe-8c4d-4ec7-9572-3245c9a37dcf None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:58.547883 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5155f5fe-8c4d-4ec7-9572-3245c9a37dcf None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:58.551219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 395/773] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:38:58.560331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f090e-014f-4462-817a-cfd726b1c0e9 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:58.560838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f090e-014f-4462-817a-cfd726b1c0e9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:58.561339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f090e-014f-4462-817a-cfd726b1c0e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:58.572750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:58.586426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de0578d-5adc-4c81-a4cb-3f124ce7be72 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:58.598233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a14f090e-014f-4462-817a-cfd726b1c0e9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:58.601184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a14f090e-014f-4462-817a-cfd726b1c0e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:58.639526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245108c1-5a36-4059-9bca-0b6abf8b27aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:58.640617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245108c1-5a36-4059-9bca-0b6abf8b27aa None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:58.641163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245108c1-5a36-4059-9bca-0b6abf8b27aa None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:58.641695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245108c1-5a36-4059-9bca-0b6abf8b27aa None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:58.642647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245108c1-5a36-4059-9bca-0b6abf8b27aa None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:58.645225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245108c1-5a36-4059-9bca-0b6abf8b27aa None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:58.671073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 379/774] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:38:58 2022] POST /identity/v3/endpoints => generated 334 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:38:58.750261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b9d6d-d27b-49c9-830f-01d9bda6a735 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:58.750961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b9d6d-d27b-49c9-830f-01d9bda6a735 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:58.751565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b9d6d-d27b-49c9-830f-01d9bda6a735 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:58.752961 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 396/775] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:58 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:38:58.753981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 380/776] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:38:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:38:58.761044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da645f9-a319-4ad2-ab2b-96ce36cb6202 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:58.761490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da645f9-a319-4ad2-ab2b-96ce36cb6202 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:58.761920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da645f9-a319-4ad2-ab2b-96ce36cb6202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:58.791689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5da645f9-a319-4ad2-ab2b-96ce36cb6202 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:38:58.793590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5da645f9-a319-4ad2-ab2b-96ce36cb6202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:38:58.929810 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 397/777] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:38:58 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:38:59.038978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b9d6d-d27b-49c9-830f-01d9bda6a735 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:59.126016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:59.164800 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b9d6d-d27b-49c9-830f-01d9bda6a735 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:59.203092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:59.203624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:59.203902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:59.204185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:59.204855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:59.206369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:59.209243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING keystone.server.flask.common [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 24 08:38:59.222866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.sql.core [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 24 08:38:59.222866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 24 08:38:59.222866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [parameters: {'user_id': 'af0a5b5ab30a419fbf9801251bceb319', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 24 08:38:59.222866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=93079) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result.read() Oct 24 08:38:59.231832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:38:59.232857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application yield resource Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.session.rollback() Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.force_reraise() Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise self.value Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application session.commit() Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 24 08:38:59.234291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._prepare_impl() Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.session.flush() Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._flush(objects) Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application compat.raise_( Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application flush_context.execute() Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rec.execute(self) Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 24 08:38:59.235378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:38:59.236827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result.read() Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:38:59.245279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [parameters: {'user_id': 'af0a5b5ab30a419fbf9801251bceb319', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:59.246395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 24 08:38:59.247268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 24 08:38:59.248546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 24 08:38:59.248546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 24 08:38:59.248546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 24 08:38:59.248546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 24 08:38:59.248546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 24 08:38:59.248546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:59.248546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 381/778] 10.222.0.39 () {60 vars in 1131 bytes} [Mon Oct 24 08:38:59 2022] POST /identity/v3/users => generated 160 bytes in 195 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:38:59.249060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da645f9-a319-4ad2-ab2b-96ce36cb6202 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:59.316607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:59.378310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da645f9-a319-4ad2-ab2b-96ce36cb6202 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:59.396413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:59.443741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:59.444875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:59.445595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:59.445980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:59.461729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:59.465116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:38:59.491317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:59.492690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 24 08:38:59.493642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:38:59.494342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 398/779] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:38:59 2022] GET /identity/v3/users/cinder => generated 84 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:59.501400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:59.528666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6f4bff-f320-4d58-a89e-402261a91f9b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:59.568829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:59.569468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:59.569752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:59.570021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] PATH_INFO: `/v3/services/image` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:59.570603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:59.572630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:59.574810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:38:59.586381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:38:59.587586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 24 08:38:59.589009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:38:59.591957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 382/780] 10.222.0.39 () {56 vars in 1105 bytes} [Mon Oct 24 08:38:59 2022] GET /identity/v3/services/image => generated 86 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:38:59.599016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:59.644156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f701d6-a1ab-406f-9af5-aaa5362a60c6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:59.680768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:38:59.697942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:59.699193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:59.699571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:59.699941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:59.700782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:59.701436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:59.703841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:59.756359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 399/781] 10.222.0.39 () {56 vars in 1101 bytes} [Mon Oct 24 08:38:59 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:59.771460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f035b7f8-783a-4f07-b183-0143c8c3d578 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:38:59.826195 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:38:59.828559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:38:59.828941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:38:59.829582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:38:59.830430 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] RBAC: Adding query filter params (name=image) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:38:59.830816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:38:59.832436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:38:59.843337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 383/782] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:59 2022] GET /identity/v3/services?name=image => generated 122 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:38:59.853558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:38:59.993665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f825993-363f-44df-9b66-756cc5d7e129 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:00.039641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26292278-8c37-46cd-a8eb-85d4f981aa09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:00.040207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26292278-8c37-46cd-a8eb-85d4f981aa09 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:00.040536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26292278-8c37-46cd-a8eb-85d4f981aa09 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:00.040809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26292278-8c37-46cd-a8eb-85d4f981aa09 None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:00.041978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26292278-8c37-46cd-a8eb-85d4f981aa09 None admin] RBAC: Adding query filter params (type=image) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:00.042297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26292278-8c37-46cd-a8eb-85d4f981aa09 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:00.043825 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26292278-8c37-46cd-a8eb-85d4f981aa09 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:00.050725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 400/783] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:38:59 2022] GET /identity/v3/services?type=image => generated 350 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:00.057230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:00.159816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:00.213618 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126d9a0e-206b-4b52-aec3-1c0d4bbd18ff None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:00.274959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41bef5e-28a4-4c09-85fd-f90ca34a5743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:00.276565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bef5e-28a4-4c09-85fd-f90ca34a5743 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:00.277458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bef5e-28a4-4c09-85fd-f90ca34a5743 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:00.277944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bef5e-28a4-4c09-85fd-f90ca34a5743 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:00.279327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41bef5e-28a4-4c09-85fd-f90ca34a5743 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:00.282364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41bef5e-28a4-4c09-85fd-f90ca34a5743 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:00.328279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 384/784] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:39:00 2022] POST /identity/v3/endpoints => generated 327 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:39:00.687969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b913f003-af77-429b-9ee1-f4f830e7868b None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:00.688749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b913f003-af77-429b-9ee1-f4f830e7868b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:00.688749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b913f003-af77-429b-9ee1-f4f830e7868b None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:00.690272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 401/785] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:00.696077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debed87-e646-4241-b5cb-e2c25a370587 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:00.696395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debed87-e646-4241-b5cb-e2c25a370587 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:00.696669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debed87-e646-4241-b5cb-e2c25a370587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:00.718266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-4debed87-e646-4241-b5cb-e2c25a370587 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:00.719796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4debed87-e646-4241-b5cb-e2c25a370587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:00.817845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 385/786] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:01.110640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b913f003-af77-429b-9ee1-f4f830e7868b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:01.163867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:01.238958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b913f003-af77-429b-9ee1-f4f830e7868b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:01.287484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d86f5e4-42fa-45d5-9881-cc21c955c712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:01.288674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86f5e4-42fa-45d5-9881-cc21c955c712 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:01.289368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86f5e4-42fa-45d5-9881-cc21c955c712 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:01.289998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86f5e4-42fa-45d5-9881-cc21c955c712 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:01.292856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d86f5e4-42fa-45d5-9881-cc21c955c712 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:01.295391 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d86f5e4-42fa-45d5-9881-cc21c955c712 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:01.297666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 402/787] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:01.305253 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4debed87-e646-4241-b5cb-e2c25a370587 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:01.382772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:01.480171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76427d8a-38aa-4d2d-a0c8-25898c05668c None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:01.480577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76427d8a-38aa-4d2d-a0c8-25898c05668c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:01.480771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76427d8a-38aa-4d2d-a0c8-25898c05668c None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:01.482447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 403/788] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:39:01.485425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4debed87-e646-4241-b5cb-e2c25a370587 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:01.489570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a04af-8cf5-4a92-8fcc-65fe99459b09 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:01.489990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a04af-8cf5-4a92-8fcc-65fe99459b09 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:01.490293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a04af-8cf5-4a92-8fcc-65fe99459b09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:01.517246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e90a04af-8cf5-4a92-8fcc-65fe99459b09 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:01.519234 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e90a04af-8cf5-4a92-8fcc-65fe99459b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:01.558739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49366a75-4d92-4bae-927f-82880bbff83b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:01.559678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49366a75-4d92-4bae-927f-82880bbff83b None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:01.560270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49366a75-4d92-4bae-927f-82880bbff83b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:01.560711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49366a75-4d92-4bae-927f-82880bbff83b None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:01.561957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49366a75-4d92-4bae-927f-82880bbff83b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:01.564902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49366a75-4d92-4bae-927f-82880bbff83b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:01.614719 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 404/789] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:39:01.624277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 386/790] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:39:01 2022] POST /identity/v3/projects => generated 311 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:39:01.934527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90a04af-8cf5-4a92-8fcc-65fe99459b09 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:01.985566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:02.086696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90a04af-8cf5-4a92-8fcc-65fe99459b09 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:02.138613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80975a8c-3fbe-4193-8dc6-378ca73c91e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:02.139355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80975a8c-3fbe-4193-8dc6-378ca73c91e0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:02.139622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80975a8c-3fbe-4193-8dc6-378ca73c91e0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:02.140067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80975a8c-3fbe-4193-8dc6-378ca73c91e0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:02.154693 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80975a8c-3fbe-4193-8dc6-378ca73c91e0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-80975a8c-3fbe-4193-8dc6-378ca73c91e0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:39:02.159991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:02.161384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 24 08:39:02.163298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:02.169362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 405/791] 10.222.0.39 () {56 vars in 1120 bytes} [Mon Oct 24 08:39:01 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:39:02.169549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49366a75-4d92-4bae-927f-82880bbff83b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:02.302595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:02.313178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49366a75-4d92-4bae-927f-82880bbff83b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:02.357369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:02.357980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:02.358251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:02.358519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:02.359235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:02.359560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:02.361131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:02.372726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 387/792] 10.222.0.39 () {56 vars in 1115 bytes} [Mon Oct 24 08:39:02 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:02.378866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80975a8c-3fbe-4193-8dc6-378ca73c91e0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:02.617678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:02.624769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80975a8c-3fbe-4193-8dc6-378ca73c91e0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:02.685303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed52234-06da-43bd-9ec3-b3a66bd764c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:02.686242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed52234-06da-43bd-9ec3-b3a66bd764c8 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:02.686617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed52234-06da-43bd-9ec3-b3a66bd764c8 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:02.686996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed52234-06da-43bd-9ec3-b3a66bd764c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:02.711380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed52234-06da-43bd-9ec3-b3a66bd764c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:02.713696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed52234-06da-43bd-9ec3-b3a66bd764c8 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-4ed52234-06da-43bd-9ec3-b3a66bd764c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:39:02.737747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:02.739389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:39:02.740965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:39:02.740965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:02.740965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:02.741907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 406/793] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 364 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:39:02.749166 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:02.873981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f94ce3-ff1b-4e64-85f7-b3b647482d41 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:02.874200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f94ce3-ff1b-4e64-85f7-b3b647482d41 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:02.874582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f94ce3-ff1b-4e64-85f7-b3b647482d41 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:02.876503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 407/794] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:02.883477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762c5cfa-5fe1-41be-868a-78dfe212d74b None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:02.884247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762c5cfa-5fe1-41be-868a-78dfe212d74b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:02.885193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762c5cfa-5fe1-41be-868a-78dfe212d74b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:02.896656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c4e8b5-8b9b-4a01-8758-5de2d23f47a9 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:02.905805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-762c5cfa-5fe1-41be-868a-78dfe212d74b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:02.907029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-762c5cfa-5fe1-41be-868a-78dfe212d74b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:02.944611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:02.945276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:02.945580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:02.945881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:02.946564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:02.946875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:02.949125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:02.964969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 388/795] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:39:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:02.970551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:03.000019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:03.006973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 408/796] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:03.122447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d891aa-95c3-4b44-84e3-7d81b7d021f7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:03.171400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:03.172573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:03.173372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:03.174055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] PATH_INFO: `/v3/users/nova` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:03.184933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:03.186636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:39:03.200691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:03.202019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:39:03.203162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.205744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 389/797] 10.222.0.39 () {56 vars in 1093 bytes} [Mon Oct 24 08:39:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:39:03.214100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762c5cfa-5fe1-41be-868a-78dfe212d74b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:03.296253 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:03.343105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:03.377319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762c5cfa-5fe1-41be-868a-78dfe212d74b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:03.398338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:03.432582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:03.438455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:03.438455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:03.438455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:03.438455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:03.439282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb7d18-c320-4c9d-9ca0-852a23135b95 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:03.439425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:03.439425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:03.463815 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 409/798] 10.222.0.39 () {56 vars in 1133 bytes} [Mon Oct 24 08:39:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:03.470778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:03.486309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:03.487038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:03.487401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:03.487761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:03.488493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:03.489981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:03.500036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.sql.core [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 24 08:39:03.500036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 24 08:39:03.500036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [parameters: {'id': '924c329bc1f64933807d35d8e781cce7', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 24 08:39:03.500036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=93079) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:39:03.507986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result.read() Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 24 08:39:03.508945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application yield resource Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application next(self.gen) Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.session.rollback() Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.force_reraise() Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise self.value Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application session.commit() Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 24 08:39:03.510354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._prepare_impl() Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.session.flush() Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._flush(objects) Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application compat.raise_( Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application flush_context.execute() Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rec.execute(self) Oct 24 08:39:03.511719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 24 08:39:03.512896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result = self._query(query) Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application conn.query(q) Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application result.read() Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 24 08:39:03.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [parameters: {'id': '924c329bc1f64933807d35d8e781cce7', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 24 08:39:03.514979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:03.515896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 390/799] 10.222.0.39 () {60 vars in 1137 bytes} [Mon Oct 24 08:39:03 2022] POST /identity/v3/domains => generated 190 bytes in 214 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:39:03.517616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:03.641055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d671685-352e-489a-9310-9f2202d5659e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:03.692443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:03.692973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:03.693253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:03.693527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:03.695532 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33927f0-2802-490b-88ea-d325d86ad96c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:03.703204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:03.710787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:03.713691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:39:03.736727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:03.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:39:03.739592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:39:03.739592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:03.739592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:03.739592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 410/800] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:39:03.747112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:03.768258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:03.769280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:03.769686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:03.770144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:03.787746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:03.790289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:03.793835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:39:03.817372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:03.819228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:39:03.820485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:39:03.820485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:03.820485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:03.824467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 391/801] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:03.828763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:03.911414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325de8a-ff45-4966-a678-8a47db1e53c6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:03.972288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:03.973174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:03.973704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:03.974145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:03.975208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:03.975733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:03.977747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:03.997055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93631304-1a21-40a3-8ef0-94331d0d49a7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:04.001720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 411/802] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:39:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:04.008827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:04.057895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70277221-0004-4d24-989b-7df277bb2554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:04.058729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70277221-0004-4d24-989b-7df277bb2554 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:04.059040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70277221-0004-4d24-989b-7df277bb2554 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:04.059358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70277221-0004-4d24-989b-7df277bb2554 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:04.060280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70277221-0004-4d24-989b-7df277bb2554 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:04.060616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70277221-0004-4d24-989b-7df277bb2554 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:04.062510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70277221-0004-4d24-989b-7df277bb2554 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:04.092173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 392/803] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:39:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:04.105310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:04.149975 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a731dbad-6814-4cea-a8de-7189bc90d778 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:04.150255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a731dbad-6814-4cea-a8de-7189bc90d778 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:04.151120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a731dbad-6814-4cea-a8de-7189bc90d778 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:04.159845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 393/804] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:39:04.161751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1758becd-86ba-47e7-a5b3-7c3612f7b6be None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:04.162023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1758becd-86ba-47e7-a5b3-7c3612f7b6be None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:04.162719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1758becd-86ba-47e7-a5b3-7c3612f7b6be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:04.189849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-1758becd-86ba-47e7-a5b3-7c3612f7b6be None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:04.190286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c4043-d996-47fb-815b-5c40849c2724 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:04.193114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1758becd-86ba-47e7-a5b3-7c3612f7b6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:04.242051 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:04.242988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:04.243314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:04.243805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:04.253650 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:04.255123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:39:04.270517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:04.271578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:04.272584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:39:04.272584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:39:04.272584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:39:04.272584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:39:04.272584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:39:04.272584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:39:04.272584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:39:04.272584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:04.276759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 412/805] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:39:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:04.279978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:04.283381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:04.322301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 394/806] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:04.443450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093cc459-88a5-4282-b62e-f1f11262a66d None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:04.462250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:04.504240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:04.505208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:04.505695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:04.506181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:04.507486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:04.508037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:04.511563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:04.534277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 413/807] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:39:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:04.540482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1758becd-86ba-47e7-a5b3-7c3612f7b6be None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:04.601293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:04.677297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1758becd-86ba-47e7-a5b3-7c3612f7b6be None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:04.718499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330fe27-a211-4f66-ae12-3f2f64857a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:04.719182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330fe27-a211-4f66-ae12-3f2f64857a52 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:04.719458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330fe27-a211-4f66-ae12-3f2f64857a52 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:04.719724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330fe27-a211-4f66-ae12-3f2f64857a52 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/4244858dfb464d029b6f6bed0491557d/roles/eb1426874ce540f2a0d2c7f30c3b87f6` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:04.722127 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330fe27-a211-4f66-ae12-3f2f64857a52 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=4244858dfb464d029b6f6bed0491557d, role_id=eb1426874ce540f2a0d2c7f30c3b87f6)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:04.723749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330fe27-a211-4f66-ae12-3f2f64857a52 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:04.740418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 395/808] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:39:04 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/4244858dfb464d029b6f6bed0491557d/roles/eb1426874ce540f2a0d2c7f30c3b87f6 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:39:04.757379 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccf3c06-1b22-4c1c-b328-c90368b6716f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:04.803110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39a2ed0-68b9-4c77-93ff-3354c59297cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:04.803895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39a2ed0-68b9-4c77-93ff-3354c59297cc None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:04.804229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39a2ed0-68b9-4c77-93ff-3354c59297cc None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:04.804505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39a2ed0-68b9-4c77-93ff-3354c59297cc None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:04.805958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39a2ed0-68b9-4c77-93ff-3354c59297cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:04.808607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39a2ed0-68b9-4c77-93ff-3354c59297cc None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:04.810618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 414/809] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:04.817268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330fe27-a211-4f66-ae12-3f2f64857a52 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:04.895065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:04.983007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330fe27-a211-4f66-ae12-3f2f64857a52 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:05.045545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ceeb45-3527-4171-9344-d5cae90e1ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:05.046463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ceeb45-3527-4171-9344-d5cae90e1ea1 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:05.046931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ceeb45-3527-4171-9344-d5cae90e1ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:05.047416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ceeb45-3527-4171-9344-d5cae90e1ea1 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:05.048607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ceeb45-3527-4171-9344-d5cae90e1ea1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:05.051210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ceeb45-3527-4171-9344-d5cae90e1ea1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:05.081693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 396/810] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:39:04 2022] POST /identity/v3/users => generated 285 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:39:07.026370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14968b3d-0c17-4961-adc8-2564c3b7d382 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:07.026930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14968b3d-0c17-4961-adc8-2564c3b7d382 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:07.027169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14968b3d-0c17-4961-adc8-2564c3b7d382 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:07.028708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 415/811] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:07.034492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92e9359-643e-4fcd-a1ff-b8296cabee7e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:07.035279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92e9359-643e-4fcd-a1ff-b8296cabee7e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:07.035279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92e9359-643e-4fcd-a1ff-b8296cabee7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:07.060028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c92e9359-643e-4fcd-a1ff-b8296cabee7e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:07.061553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c92e9359-643e-4fcd-a1ff-b8296cabee7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:07.143448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 397/812] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:07.417595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14968b3d-0c17-4961-adc8-2564c3b7d382 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:07.445113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f99d5b-ba9b-4f3d-adb7-a5874ffc43e1 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:07.445608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f99d5b-ba9b-4f3d-adb7-a5874ffc43e1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:07.446856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f99d5b-ba9b-4f3d-adb7-a5874ffc43e1 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:07.449796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 398/813] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:07 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:07.458636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87e51ec-295e-49ca-9aaf-fa2ea46b6b2f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:07.459115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87e51ec-295e-49ca-9aaf-fa2ea46b6b2f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:07.459599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87e51ec-295e-49ca-9aaf-fa2ea46b6b2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:07.468722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:07.485495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-d87e51ec-295e-49ca-9aaf-fa2ea46b6b2f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:07.486710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d87e51ec-295e-49ca-9aaf-fa2ea46b6b2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:07.528858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:07.545110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14968b3d-0c17-4961-adc8-2564c3b7d382 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:07.582866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3f6327-c621-42a5-890c-6a2bbc1f2674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:07.583550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f6327-c621-42a5-890c-6a2bbc1f2674 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:07.583847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f6327-c621-42a5-890c-6a2bbc1f2674 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:07.584115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f6327-c621-42a5-890c-6a2bbc1f2674 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:07.595795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3f6327-c621-42a5-890c-6a2bbc1f2674 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-7d3f6327-c621-42a5-890c-6a2bbc1f2674 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:39:07.600913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:07.601949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:07.602885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 416/814] 10.222.0.39 () {56 vars in 1120 bytes} [Mon Oct 24 08:39:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:39:07.607625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 399/815] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:39:07.610909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3f6327-c621-42a5-890c-6a2bbc1f2674 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:07.743646 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:07.748274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3f6327-c621-42a5-890c-6a2bbc1f2674 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:07.787256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:07.787935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:07.788270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:07.788596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:07.789394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:07.789774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:07.791367 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:07.801023 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 417/816] 10.222.0.39 () {56 vars in 1115 bytes} [Mon Oct 24 08:39:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:07.808825 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87e51ec-295e-49ca-9aaf-fa2ea46b6b2f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:07.890580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:07.922958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87e51ec-295e-49ca-9aaf-fa2ea46b6b2f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:07.976567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:07.978601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:07.979210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:07.979777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:07.995594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:07.997082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:39:08.016211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:08.017256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:39:08.018332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:39:08.018332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:08.018332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:08.018332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 400/817] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:08.024413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:08.043856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:08.049400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9245f0bc-f84e-4857-aba2-6deba616db83 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.093610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d0ca7f-ed51-4f1b-976e-b144969cba02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.094303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d0ca7f-ed51-4f1b-976e-b144969cba02 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.094581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d0ca7f-ed51-4f1b-976e-b144969cba02 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.094945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d0ca7f-ed51-4f1b-976e-b144969cba02 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.108539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d0ca7f-ed51-4f1b-976e-b144969cba02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-e1d0ca7f-ed51-4f1b-976e-b144969cba02 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:39:08.113731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:08.115013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:08.117071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 418/818] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:39:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:39:08.121892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d0ca7f-ed51-4f1b-976e-b144969cba02 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:08.150662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:08.155672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c26380-0ae6-43fd-9ba3-a5f1c152d1f8 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.204233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.205003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.205343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.205876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.206789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:08.207163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:08.208916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:08.223846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 401/819] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:39:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:08.233117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:08.255413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d0ca7f-ed51-4f1b-976e-b144969cba02 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.295065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.295676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.295949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.296242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.296941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:08.297235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:08.298600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:08.309872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 419/820] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:39:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:08.317298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:08.321073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:08.363528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d002f5-2990-43a8-adc1-03b3431cdd66 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.406050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.406936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.407570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.408055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] PATH_INFO: `/v3/users/nova` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.412666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:08.422033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:08.424307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:08.435196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc019c62-7348-4462-961e-f5f91dfd91bf None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:39:08.442003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:08.443088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:08.444356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 402/821] 10.222.0.39 () {56 vars in 1093 bytes} [Mon Oct 24 08:39:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:08.453159 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:08.488193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1e7829-e8f2-4cac-853f-f7e7ab4e812b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.488833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e7829-e8f2-4cac-853f-f7e7ab4e812b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.489111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e7829-e8f2-4cac-853f-f7e7ab4e812b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.489381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e7829-e8f2-4cac-853f-f7e7ab4e812b None admin] PATH_INFO: `/v3/users/846372e3db574fff814cf4ed616565f2` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.508926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1e7829-e8f2-4cac-853f-f7e7ab4e812b None admin] RBAC: Authorizing `identity:get_user(user_id=846372e3db574fff814cf4ed616565f2)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:08.511652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1e7829-e8f2-4cac-853f-f7e7ab4e812b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:08.514270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 420/822] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:08 2022] GET /identity/v3/users/846372e3db574fff814cf4ed616565f2 => generated 285 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:08.520263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1e7829-e8f2-4cac-853f-f7e7ab4e812b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:08.588471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cfda87-50c4-43fe-871c-e2a4d091fa88 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.633401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:08.639964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.640612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.640892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.641162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.641920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:08.642228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:08.643721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:08.669954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1e7829-e8f2-4cac-853f-f7e7ab4e812b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.673749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 403/823] 10.222.0.39 () {56 vars in 1133 bytes} [Mon Oct 24 08:39:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:08.679915 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:08.715654 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72c6c21-1c2c-4f50-97eb-f0909a5be7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.716366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72c6c21-1c2c-4f50-97eb-f0909a5be7b0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.716640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72c6c21-1c2c-4f50-97eb-f0909a5be7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.716911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72c6c21-1c2c-4f50-97eb-f0909a5be7b0 None admin] PATH_INFO: `/v3/projects/aced97baba3a4f3b9c564684cd64b2ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.718151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72c6c21-1c2c-4f50-97eb-f0909a5be7b0 None admin] RBAC: Authorizing `identity:get_project(project_id=aced97baba3a4f3b9c564684cd64b2ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:08.719610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72c6c21-1c2c-4f50-97eb-f0909a5be7b0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:08.721501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 421/824] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:39:08 2022] GET /identity/v3/projects/aced97baba3a4f3b9c564684cd64b2ba => generated 311 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:08.728182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72c6c21-1c2c-4f50-97eb-f0909a5be7b0 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:08.741226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:08.818321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3217fd31-3198-4219-8fb2-291ceb61d72b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.865066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fb442a-1c7e-4c57-b3ec-48b1d3acbc36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.865789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fb442a-1c7e-4c57-b3ec-48b1d3acbc36 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.866081 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fb442a-1c7e-4c57-b3ec-48b1d3acbc36 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.866438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fb442a-1c7e-4c57-b3ec-48b1d3acbc36 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.867989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72c6c21-1c2c-4f50-97eb-f0909a5be7b0 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:08.911343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256984f3-728a-4cd8-b50e-5b6519f46a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:08.912066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256984f3-728a-4cd8-b50e-5b6519f46a00 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:08.912389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256984f3-728a-4cd8-b50e-5b6519f46a00 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:08.912702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256984f3-728a-4cd8-b50e-5b6519f46a00 None admin] PATH_INFO: `/v3/projects/aced97baba3a4f3b9c564684cd64b2ba/users/846372e3db574fff814cf4ed616565f2/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:08.914872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256984f3-728a-4cd8-b50e-5b6519f46a00 None admin] RBAC: Authorizing `identity:create_grant(project_id=aced97baba3a4f3b9c564684cd64b2ba, user_id=846372e3db574fff814cf4ed616565f2, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:08.917254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256984f3-728a-4cd8-b50e-5b6519f46a00 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:08.934322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 422/825] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:39:08 2022] PUT /identity/v3/projects/aced97baba3a4f3b9c564684cd64b2ba/users/846372e3db574fff814cf4ed616565f2/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:39:08.941916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:08.956580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fb442a-1c7e-4c57-b3ec-48b1d3acbc36 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:08.958124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fb442a-1c7e-4c57-b3ec-48b1d3acbc36 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:09.084169 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 404/826] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:39:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 24 08:39:09.090501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256984f3-728a-4cd8-b50e-5b6519f46a00 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:09.169075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:09.205603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256984f3-728a-4cd8-b50e-5b6519f46a00 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:09.241945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:09.242581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:09.242875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:09.243148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:09.255183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:09.256731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:39:09.273073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:09.274421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:39:09.275911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:39:09.275911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:39:09.275911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:09.275911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 423/827] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:39:09.283828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fb442a-1c7e-4c57-b3ec-48b1d3acbc36 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:09.409545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:09.422616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fb442a-1c7e-4c57-b3ec-48b1d3acbc36 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:09.472569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:09.473220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:09.473503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:09.473777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:09.474445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:09.474752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:09.476188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:09.491333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 405/828] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:39:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:09.497683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:09.632489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e789d4ab-044f-474b-94ad-9ea6ef43bc72 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:09.657096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:09.673871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:09.674448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:09.674728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:09.674998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:09.683966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:09.685528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:39:09.698993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:09.700228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:09.701656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 424/829] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:39:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:09.706575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:09.825292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2234e309-40d5-4820-a958-4267948986a2 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:09.864559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7c688f-e336-4bb0-9840-23b5938ad7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:09.865174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7c688f-e336-4bb0-9840-23b5938ad7f1 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:09.865452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7c688f-e336-4bb0-9840-23b5938ad7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:09.865721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7c688f-e336-4bb0-9840-23b5938ad7f1 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:09.866436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7c688f-e336-4bb0-9840-23b5938ad7f1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:09.866744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7c688f-e336-4bb0-9840-23b5938ad7f1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:09.868260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7c688f-e336-4bb0-9840-23b5938ad7f1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:09.882664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 406/830] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:39:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:09.889737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:10.027584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1c1f75-a948-43e9-ab7d-4ed9300a13f3 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:10.065718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1690c168-ba13-4c4a-8fd9-af33cc6202eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:10.066310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690c168-ba13-4c4a-8fd9-af33cc6202eb None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:10.066583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690c168-ba13-4c4a-8fd9-af33cc6202eb None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:10.066849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690c168-ba13-4c4a-8fd9-af33cc6202eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:10.067558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1690c168-ba13-4c4a-8fd9-af33cc6202eb None admin] RBAC: Adding query filter params (role.id=eb1426874ce540f2a0d2c7f30c3b87f6, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=4244858dfb464d029b6f6bed0491557d) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:10.067858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1690c168-ba13-4c4a-8fd9-af33cc6202eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:10.069302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1690c168-ba13-4c4a-8fd9-af33cc6202eb None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:10.077828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 425/831] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:39:09 2022] GET /identity/v3/role_assignments?user.id=4244858dfb464d029b6f6bed0491557d&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=eb1426874ce540f2a0d2c7f30c3b87f6 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:11.203771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590c1641-7876-4a73-a734-028566b8ee47 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:11.204355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590c1641-7876-4a73-a734-028566b8ee47 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:11.204452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590c1641-7876-4a73-a734-028566b8ee47 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:11.205830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 407/832] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:11.210958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bbd832-fd96-47de-967e-584abe9e45e4 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:11.211337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bbd832-fd96-47de-967e-584abe9e45e4 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:11.211508 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bbd832-fd96-47de-967e-584abe9e45e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:11.229853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b4bbd832-fd96-47de-967e-584abe9e45e4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:11.231015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b4bbd832-fd96-47de-967e-584abe9e45e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:11.275356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:11.301615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 426/833] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:11.571588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590c1641-7876-4a73-a734-028566b8ee47 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:11.586716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:11.684301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590c1641-7876-4a73-a734-028566b8ee47 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:11.717579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c77d94-9224-42af-a1d8-37d470632fb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:11.718191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c77d94-9224-42af-a1d8-37d470632fb4 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:11.718473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c77d94-9224-42af-a1d8-37d470632fb4 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:11.718748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c77d94-9224-42af-a1d8-37d470632fb4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:11.729924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c77d94-9224-42af-a1d8-37d470632fb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-45c77d94-9224-42af-a1d8-37d470632fb4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:39:11.734744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:11.735855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:11.736978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 408/834] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:39:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:11.741963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bbd832-fd96-47de-967e-584abe9e45e4 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:11.839214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bbd832-fd96-47de-967e-584abe9e45e4 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:11.872019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:11.872645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:11.872927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:11.873199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:11.873850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:11.874155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:11.875505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:11.883310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 427/835] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:39:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:11.889820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c77d94-9224-42af-a1d8-37d470632fb4 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:11.959018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:11.996898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c77d94-9224-42af-a1d8-37d470632fb4 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:12.032348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8370d2f2-6e9e-48bb-8de4-7059063e4c10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:12.033069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8370d2f2-6e9e-48bb-8de4-7059063e4c10 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:12.033407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8370d2f2-6e9e-48bb-8de4-7059063e4c10 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:12.033712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8370d2f2-6e9e-48bb-8de4-7059063e4c10 None admin] PATH_INFO: `/v3/users/846372e3db574fff814cf4ed616565f2` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:12.034918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8370d2f2-6e9e-48bb-8de4-7059063e4c10 None admin] RBAC: Authorizing `identity:get_user(user_id=846372e3db574fff814cf4ed616565f2)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:12.036450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8370d2f2-6e9e-48bb-8de4-7059063e4c10 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:12.038172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 409/836] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:11 2022] GET /identity/v3/users/846372e3db574fff814cf4ed616565f2 => generated 285 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:12.043986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:12.079096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:12.145770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657ae3a-06a5-49c5-86da-dc6c32aeed26 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:12.179421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e79db7-de6f-4e71-9861-fdd9916860d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:12.180022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e79db7-de6f-4e71-9861-fdd9916860d7 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:12.180336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e79db7-de6f-4e71-9861-fdd9916860d7 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:12.180610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e79db7-de6f-4e71-9861-fdd9916860d7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:12.250280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e79db7-de6f-4e71-9861-fdd9916860d7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:12.251873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e79db7-de6f-4e71-9861-fdd9916860d7 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:12.285363 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:12.348635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 428/837] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:39:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 24 08:39:12.355561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8370d2f2-6e9e-48bb-8de4-7059063e4c10 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:12.455655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8370d2f2-6e9e-48bb-8de4-7059063e4c10 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:12.489874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6449ce2-bea1-4e58-bd17-51fc310e1513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:12.490489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6449ce2-bea1-4e58-bd17-51fc310e1513 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:12.490767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6449ce2-bea1-4e58-bd17-51fc310e1513 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:12.491033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6449ce2-bea1-4e58-bd17-51fc310e1513 None admin] PATH_INFO: `/v3/projects/aced97baba3a4f3b9c564684cd64b2ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:12.492117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6449ce2-bea1-4e58-bd17-51fc310e1513 None admin] RBAC: Authorizing `identity:get_project(project_id=aced97baba3a4f3b9c564684cd64b2ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:12.493573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6449ce2-bea1-4e58-bd17-51fc310e1513 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:12.496024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 410/838] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:39:12 2022] GET /identity/v3/projects/aced97baba3a4f3b9c564684cd64b2ba => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:12.502142 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e79db7-de6f-4e71-9861-fdd9916860d7 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:12.610259 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e79db7-de6f-4e71-9861-fdd9916860d7 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:12.648549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d75513-c50b-4463-b042-9b558ec597c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:12.649137 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d75513-c50b-4463-b042-9b558ec597c0 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:12.649412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d75513-c50b-4463-b042-9b558ec597c0 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:12.649681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d75513-c50b-4463-b042-9b558ec597c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:12.651018 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d75513-c50b-4463-b042-9b558ec597c0 None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=aced97baba3a4f3b9c564684cd64b2ba, user.id=846372e3db574fff814cf4ed616565f2) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:12.651324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d75513-c50b-4463-b042-9b558ec597c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:12.652754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d75513-c50b-4463-b042-9b558ec597c0 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:12.655501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:12.659527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 429/839] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:39:12 2022] GET /identity/v3/role_assignments?user.id=846372e3db574fff814cf4ed616565f2&scope.project.id=aced97baba3a4f3b9c564684cd64b2ba&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:14.950028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb27869-7e16-4f73-9f1e-815ac304b2c7 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:14.950559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb27869-7e16-4f73-9f1e-815ac304b2c7 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:14.950652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb27869-7e16-4f73-9f1e-815ac304b2c7 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:14.952202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 411/840] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:14.957921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3d541e-c920-4140-9222-f60534b833bf None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:14.958336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3d541e-c920-4140-9222-f60534b833bf None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:14.958457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3d541e-c920-4140-9222-f60534b833bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:14.977156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6d3d541e-c920-4140-9222-f60534b833bf None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:14.978852 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6d3d541e-c920-4140-9222-f60534b833bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:15.045256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 430/841] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:15.318970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb27869-7e16-4f73-9f1e-815ac304b2c7 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:15.328979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:15.421282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb27869-7e16-4f73-9f1e-815ac304b2c7 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:15.455063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724c17e-25be-4424-baf9-761cd61fb228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:15.455678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724c17e-25be-4424-baf9-761cd61fb228 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:15.455954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724c17e-25be-4424-baf9-761cd61fb228 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:15.456251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724c17e-25be-4424-baf9-761cd61fb228 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:15.457345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9724c17e-25be-4424-baf9-761cd61fb228 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:15.458764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9724c17e-25be-4424-baf9-761cd61fb228 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:15.460361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 412/842] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:15.466855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3d541e-c920-4140-9222-f60534b833bf None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:15.571772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3d541e-c920-4140-9222-f60534b833bf None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:15.600667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:15.607684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a4ec3d-3404-4529-ada5-b3f024c653f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:15.608320 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a4ec3d-3404-4529-ada5-b3f024c653f3 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a4ec3d-3404-4529-ada5-b3f024c653f3 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:15.608866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a4ec3d-3404-4529-ada5-b3f024c653f3 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:15.609521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a4ec3d-3404-4529-ada5-b3f024c653f3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:15.610942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a4ec3d-3404-4529-ada5-b3f024c653f3 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:15.632600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 431/843] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:39:15 2022] POST /identity/v3/users => generated 286 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:39:17.867725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2d755-ebfe-49e3-ae2c-14951f7dac00 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:17.869783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2d755-ebfe-49e3-ae2c-14951f7dac00 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:17.869783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2d755-ebfe-49e3-ae2c-14951f7dac00 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:17.870426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 413/844] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:17.876228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5807da7-c1b2-43e5-84b4-5d34cf2d8990 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:17.876675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5807da7-c1b2-43e5-84b4-5d34cf2d8990 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:17.876808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5807da7-c1b2-43e5-84b4-5d34cf2d8990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:17.896384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b5807da7-c1b2-43e5-84b4-5d34cf2d8990 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:17.897793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b5807da7-c1b2-43e5-84b4-5d34cf2d8990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:17.975525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 432/845] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:18.261980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b2d755-ebfe-49e3-ae2c-14951f7dac00 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:18.344636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:18.376352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b2d755-ebfe-49e3-ae2c-14951f7dac00 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:18.416966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bdb05b-5f7a-4502-8d20-308019ef9578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:18.417958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bdb05b-5f7a-4502-8d20-308019ef9578 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:18.418419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bdb05b-5f7a-4502-8d20-308019ef9578 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:18.418891 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bdb05b-5f7a-4502-8d20-308019ef9578 None admin] PATH_INFO: `/v3/roles/9b956c43ef364a70bf8263ba4314dce7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:18.420588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bdb05b-5f7a-4502-8d20-308019ef9578 None admin] RBAC: Authorizing `identity:get_role(role_id=9b956c43ef364a70bf8263ba4314dce7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:18.421964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bdb05b-5f7a-4502-8d20-308019ef9578 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:18.423312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 414/846] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:18 2022] GET /identity/v3/roles/9b956c43ef364a70bf8263ba4314dce7 => generated 223 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:18.429205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5807da7-c1b2-43e5-84b4-5d34cf2d8990 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:18.539828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5807da7-c1b2-43e5-84b4-5d34cf2d8990 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:18.559387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:18.584518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd41aaa-7198-45c8-8c03-f6f03ff05bf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:18.585207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd41aaa-7198-45c8-8c03-f6f03ff05bf9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:18.585466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd41aaa-7198-45c8-8c03-f6f03ff05bf9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:18.585711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd41aaa-7198-45c8-8c03-f6f03ff05bf9 None admin] PATH_INFO: `/v3/users/cfae3ab3897646dbb52a2ab2832db75d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:18.600785 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd41aaa-7198-45c8-8c03-f6f03ff05bf9 None admin] RBAC: Authorizing `identity:get_user(user_id=cfae3ab3897646dbb52a2ab2832db75d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:18.602235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd41aaa-7198-45c8-8c03-f6f03ff05bf9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:18.604102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 433/847] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:18 2022] GET /identity/v3/users/cfae3ab3897646dbb52a2ab2832db75d => generated 286 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:18.612495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bdb05b-5f7a-4502-8d20-308019ef9578 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:18.729854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bdb05b-5f7a-4502-8d20-308019ef9578 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:18.767329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb25674b-cce1-468a-9b9a-804ac6f09c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:18.767959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25674b-cce1-468a-9b9a-804ac6f09c45 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:18.768268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25674b-cce1-468a-9b9a-804ac6f09c45 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:18.768542 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25674b-cce1-468a-9b9a-804ac6f09c45 None admin] PATH_INFO: `/v3/projects/aced97baba3a4f3b9c564684cd64b2ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:18.769666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb25674b-cce1-468a-9b9a-804ac6f09c45 None admin] RBAC: Authorizing `identity:get_project(project_id=aced97baba3a4f3b9c564684cd64b2ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:18.771300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb25674b-cce1-468a-9b9a-804ac6f09c45 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:18.773039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 415/848] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:39:18 2022] GET /identity/v3/projects/aced97baba3a4f3b9c564684cd64b2ba => generated 311 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:18.779816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd41aaa-7198-45c8-8c03-f6f03ff05bf9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:18.886534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd41aaa-7198-45c8-8c03-f6f03ff05bf9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:18.921777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87a13bf-8c3e-4dc1-abfc-24de3d488710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:18.922488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a13bf-8c3e-4dc1-abfc-24de3d488710 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:18.922972 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a13bf-8c3e-4dc1-abfc-24de3d488710 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:18.923481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a13bf-8c3e-4dc1-abfc-24de3d488710 None admin] PATH_INFO: `/v3/projects/aced97baba3a4f3b9c564684cd64b2ba/users/cfae3ab3897646dbb52a2ab2832db75d/roles/9b956c43ef364a70bf8263ba4314dce7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:18.925902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87a13bf-8c3e-4dc1-abfc-24de3d488710 None admin] RBAC: Authorizing `identity:create_grant(project_id=aced97baba3a4f3b9c564684cd64b2ba, user_id=cfae3ab3897646dbb52a2ab2832db75d, role_id=9b956c43ef364a70bf8263ba4314dce7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:18.927418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87a13bf-8c3e-4dc1-abfc-24de3d488710 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:18.945388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 434/849] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:39:18 2022] PUT /identity/v3/projects/aced97baba3a4f3b9c564684cd64b2ba/users/cfae3ab3897646dbb52a2ab2832db75d/roles/9b956c43ef364a70bf8263ba4314dce7 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:39:21.239421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e194f-8483-4a24-8bd2-865f260032ae None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:21.239995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e194f-8483-4a24-8bd2-865f260032ae None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:21.239995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e194f-8483-4a24-8bd2-865f260032ae None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:21.241382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 416/850] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:21.247538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dbc47a-14a1-40da-a5d2-b33ec2ada2cf None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:21.247961 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dbc47a-14a1-40da-a5d2-b33ec2ada2cf None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:21.248070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dbc47a-14a1-40da-a5d2-b33ec2ada2cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:21.267264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-32dbc47a-14a1-40da-a5d2-b33ec2ada2cf None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:21.268475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-32dbc47a-14a1-40da-a5d2-b33ec2ada2cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:21.293873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:21.334648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 435/851] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:21.616991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e194f-8483-4a24-8bd2-865f260032ae None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:21.631518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:21.721963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e194f-8483-4a24-8bd2-865f260032ae None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:21.755593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16bda0-2a06-495c-b4a9-c8b1c701d8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:21.756228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16bda0-2a06-495c-b4a9-c8b1c701d8d7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:21.756506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16bda0-2a06-495c-b4a9-c8b1c701d8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:21.756774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16bda0-2a06-495c-b4a9-c8b1c701d8d7 None admin] PATH_INFO: `/v3/roles/9b956c43ef364a70bf8263ba4314dce7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:21.758137 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a16bda0-2a06-495c-b4a9-c8b1c701d8d7 None admin] RBAC: Authorizing `identity:get_role(role_id=9b956c43ef364a70bf8263ba4314dce7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:21.760028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a16bda0-2a06-495c-b4a9-c8b1c701d8d7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:21.761903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 417/852] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:21 2022] GET /identity/v3/roles/9b956c43ef364a70bf8263ba4314dce7 => generated 223 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:21.767791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dbc47a-14a1-40da-a5d2-b33ec2ada2cf None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:21.870584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dbc47a-14a1-40da-a5d2-b33ec2ada2cf None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:21.907447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3517035b-7c40-4d8b-bcb4-3e46f399de60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:21.908075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3517035b-7c40-4d8b-bcb4-3e46f399de60 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:21.908385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3517035b-7c40-4d8b-bcb4-3e46f399de60 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:21.908659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3517035b-7c40-4d8b-bcb4-3e46f399de60 None admin] PATH_INFO: `/v3/users/cfae3ab3897646dbb52a2ab2832db75d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:21.909799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3517035b-7c40-4d8b-bcb4-3e46f399de60 None admin] RBAC: Authorizing `identity:get_user(user_id=cfae3ab3897646dbb52a2ab2832db75d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:21.911219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3517035b-7c40-4d8b-bcb4-3e46f399de60 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:21.912856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 436/853] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:21 2022] GET /identity/v3/users/cfae3ab3897646dbb52a2ab2832db75d => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:21.919047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16bda0-2a06-495c-b4a9-c8b1c701d8d7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:22.002570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:22.025779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16bda0-2a06-495c-b4a9-c8b1c701d8d7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:22.061323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d894781-d98d-44a1-83e1-3abeedd24047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:22.061925 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d894781-d98d-44a1-83e1-3abeedd24047 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:22.062209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d894781-d98d-44a1-83e1-3abeedd24047 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:22.062486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d894781-d98d-44a1-83e1-3abeedd24047 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:22.135264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d894781-d98d-44a1-83e1-3abeedd24047 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:22.136844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d894781-d98d-44a1-83e1-3abeedd24047 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:22.216728 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:22.240090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 418/854] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:39:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 24 08:39:22.246565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3517035b-7c40-4d8b-bcb4-3e46f399de60 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:22.277806 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:22.350240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3517035b-7c40-4d8b-bcb4-3e46f399de60 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:22.387931 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7616710-9d8b-4a1f-a539-2110f0f26183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:22.388510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7616710-9d8b-4a1f-a539-2110f0f26183 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:22.388791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7616710-9d8b-4a1f-a539-2110f0f26183 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:22.389064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7616710-9d8b-4a1f-a539-2110f0f26183 None admin] PATH_INFO: `/v3/projects/aced97baba3a4f3b9c564684cd64b2ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:22.390196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7616710-9d8b-4a1f-a539-2110f0f26183 None admin] RBAC: Authorizing `identity:get_project(project_id=aced97baba3a4f3b9c564684cd64b2ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:22.391628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7616710-9d8b-4a1f-a539-2110f0f26183 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:22.393327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 437/855] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:39:22 2022] GET /identity/v3/projects/aced97baba3a4f3b9c564684cd64b2ba => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:22.399653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d894781-d98d-44a1-83e1-3abeedd24047 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:22.506910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d894781-d98d-44a1-83e1-3abeedd24047 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:22.545835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd4579f-4f54-4de4-953b-1929a6dfa6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:22.546601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd4579f-4f54-4de4-953b-1929a6dfa6ca None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:22.546960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd4579f-4f54-4de4-953b-1929a6dfa6ca None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:22.547272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd4579f-4f54-4de4-953b-1929a6dfa6ca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:22.548182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd4579f-4f54-4de4-953b-1929a6dfa6ca None admin] RBAC: Adding query filter params (role.id=9b956c43ef364a70bf8263ba4314dce7, scope.project.id=aced97baba3a4f3b9c564684cd64b2ba, user.id=cfae3ab3897646dbb52a2ab2832db75d) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:22.548578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd4579f-4f54-4de4-953b-1929a6dfa6ca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:22.550319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd4579f-4f54-4de4-953b-1929a6dfa6ca None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:22.557812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 419/856] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:39:22 2022] GET /identity/v3/role_assignments?user.id=cfae3ab3897646dbb52a2ab2832db75d&scope.project.id=aced97baba3a4f3b9c564684cd64b2ba&role.id=9b956c43ef364a70bf8263ba4314dce7 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:24.823978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed0249-1813-4c56-9195-0d873ae172c9 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:24.824607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed0249-1813-4c56-9195-0d873ae172c9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:24.824607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed0249-1813-4c56-9195-0d873ae172c9 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:24.825888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 438/857] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:24.831593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b18a6c-244b-49ac-b02b-ff5b142682bf None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:24.832171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b18a6c-244b-49ac-b02b-ff5b142682bf None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:24.832171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b18a6c-244b-49ac-b02b-ff5b142682bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:24.851005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-60b18a6c-244b-49ac-b02b-ff5b142682bf None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:24.852230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-60b18a6c-244b-49ac-b02b-ff5b142682bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:24.888844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:24.921912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 420/858] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:25.192543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ed0249-1813-4c56-9195-0d873ae172c9 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:25.292992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:25.297670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ed0249-1813-4c56-9195-0d873ae172c9 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:25.333157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c401bbff-edf5-4138-b4c2-4053cd35272d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:25.333791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401bbff-edf5-4138-b4c2-4053cd35272d None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:25.334072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401bbff-edf5-4138-b4c2-4053cd35272d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:25.334346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401bbff-edf5-4138-b4c2-4053cd35272d None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:25.335502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c401bbff-edf5-4138-b4c2-4053cd35272d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:25.336951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c401bbff-edf5-4138-b4c2-4053cd35272d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:25.339145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 439/859] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:25.345212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b18a6c-244b-49ac-b02b-ff5b142682bf None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:25.454071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b18a6c-244b-49ac-b02b-ff5b142682bf None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:25.487735 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1539ff-ebef-4226-a27a-709c2514868c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:25.488358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1539ff-ebef-4226-a27a-709c2514868c None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:25.488636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1539ff-ebef-4226-a27a-709c2514868c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:25.488903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1539ff-ebef-4226-a27a-709c2514868c None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:25.489542 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1539ff-ebef-4226-a27a-709c2514868c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:25.490962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1539ff-ebef-4226-a27a-709c2514868c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:25.519567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 421/860] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:39:25 2022] POST /identity/v3/projects => generated 311 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:39:27.737015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7cf3a2-3e6e-4ea4-aaad-c2bf03e553ee None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:27.737630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7cf3a2-3e6e-4ea4-aaad-c2bf03e553ee None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:27.737630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7cf3a2-3e6e-4ea4-aaad-c2bf03e553ee None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:27.738879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 440/861] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:39:27.744838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c930769-37e4-4287-adc2-649a45a8590b None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:27.745656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c930769-37e4-4287-adc2-649a45a8590b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:27.745765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c930769-37e4-4287-adc2-649a45a8590b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:27.765498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7c930769-37e4-4287-adc2-649a45a8590b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:27.766694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7c930769-37e4-4287-adc2-649a45a8590b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:27.785859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:27.838332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 422/862] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:28.111643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7cf3a2-3e6e-4ea4-aaad-c2bf03e553ee None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:28.211192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7cf3a2-3e6e-4ea4-aaad-c2bf03e553ee None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:28.244780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7bbe69-a2b7-4a9e-ab72-ba1cb2f7d129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:28.245596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7bbe69-a2b7-4a9e-ab72-ba1cb2f7d129 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:28.245877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7bbe69-a2b7-4a9e-ab72-ba1cb2f7d129 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:28.246149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7bbe69-a2b7-4a9e-ab72-ba1cb2f7d129 None admin] PATH_INFO: `/v3/domains/default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:28.247257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7bbe69-a2b7-4a9e-ab72-ba1cb2f7d129 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:28.248708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7bbe69-a2b7-4a9e-ab72-ba1cb2f7d129 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:28.250269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 441/863] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:39:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:28.256059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c930769-37e4-4287-adc2-649a45a8590b None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:28.351822 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c930769-37e4-4287-adc2-649a45a8590b None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:28.387823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b6ca04-05db-4787-9a6b-5cfa2743d0ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:28.388452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b6ca04-05db-4787-9a6b-5cfa2743d0ec None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:28.388732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b6ca04-05db-4787-9a6b-5cfa2743d0ec None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:28.389004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b6ca04-05db-4787-9a6b-5cfa2743d0ec None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:28.389658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b6ca04-05db-4787-9a6b-5cfa2743d0ec None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:28.391083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b6ca04-05db-4787-9a6b-5cfa2743d0ec None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:28.412844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 423/864] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:39:28 2022] POST /identity/v3/users => generated 286 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:39:30.593210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca396a6-a83f-4b39-b22e-a7e543da1e10 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:30.593767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca396a6-a83f-4b39-b22e-a7e543da1e10 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:30.593767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca396a6-a83f-4b39-b22e-a7e543da1e10 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:30.595083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 442/865] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:30.600328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da348e-3a72-4b0e-9729-30fec26cbf56 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:30.600750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da348e-3a72-4b0e-9729-30fec26cbf56 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:30.600880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da348e-3a72-4b0e-9729-30fec26cbf56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:30.619841 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-51da348e-3a72-4b0e-9729-30fec26cbf56 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:30.621319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-51da348e-3a72-4b0e-9729-30fec26cbf56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:30.623951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:30.685920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 424/866] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:30.955346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca396a6-a83f-4b39-b22e-a7e543da1e10 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:30.993362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:31.059227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca396a6-a83f-4b39-b22e-a7e543da1e10 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:31.094650 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580279a7-8bd0-4fd8-b217-5f266af4c357 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:31.095267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580279a7-8bd0-4fd8-b217-5f266af4c357 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:31.095547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580279a7-8bd0-4fd8-b217-5f266af4c357 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:31.095822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580279a7-8bd0-4fd8-b217-5f266af4c357 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:31.106864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580279a7-8bd0-4fd8-b217-5f266af4c357 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-580279a7-8bd0-4fd8-b217-5f266af4c357 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:39:31.111671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:31.112880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:31.114057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 443/867] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:39:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:31.119001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51da348e-3a72-4b0e-9729-30fec26cbf56 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:31.219983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51da348e-3a72-4b0e-9729-30fec26cbf56 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:31.255633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:31.256266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:31.256552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:31.256826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:31.257486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:31.257790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:31.259156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:31.268437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 425/868] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:39:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:31.274382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580279a7-8bd0-4fd8-b217-5f266af4c357 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:31.373717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:31.380112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580279a7-8bd0-4fd8-b217-5f266af4c357 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:31.414348 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0819f4-3d14-47b5-9f8c-a932c83902a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:31.414967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0819f4-3d14-47b5-9f8c-a932c83902a6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:31.415244 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0819f4-3d14-47b5-9f8c-a932c83902a6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:31.415516 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0819f4-3d14-47b5-9f8c-a932c83902a6 None admin] PATH_INFO: `/v3/users/5812de70afbe44778a0321486e3e7ec8` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:31.429087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0819f4-3d14-47b5-9f8c-a932c83902a6 None admin] RBAC: Authorizing `identity:get_user(user_id=5812de70afbe44778a0321486e3e7ec8)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:31.430529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0819f4-3d14-47b5-9f8c-a932c83902a6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:31.432383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 444/869] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:31 2022] GET /identity/v3/users/5812de70afbe44778a0321486e3e7ec8 => generated 286 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:31.438273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:31.441182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:31.542532 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fab0c7-8d7f-4b2e-a80f-c0947b936102 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:31.577871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dff5ed-e9a5-448c-a22d-2506dbd69164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:31.578495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dff5ed-e9a5-448c-a22d-2506dbd69164 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:31.578772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dff5ed-e9a5-448c-a22d-2506dbd69164 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:31.579041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dff5ed-e9a5-448c-a22d-2506dbd69164 None admin] PATH_INFO: `/v3/projects/c2159628109c41eabb560834d45497f4` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:31.580182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dff5ed-e9a5-448c-a22d-2506dbd69164 None admin] RBAC: Authorizing `identity:get_project(project_id=c2159628109c41eabb560834d45497f4)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:31.581601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dff5ed-e9a5-448c-a22d-2506dbd69164 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:31.583224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 426/870] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:39:31 2022] GET /identity/v3/projects/c2159628109c41eabb560834d45497f4 => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:31.589892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0819f4-3d14-47b5-9f8c-a932c83902a6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:31.691063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0819f4-3d14-47b5-9f8c-a932c83902a6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:31.725083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738cb3e0-d30f-40c8-8d95-2877f43b404d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:31.725732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cb3e0-d30f-40c8-8d95-2877f43b404d None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:31.726013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cb3e0-d30f-40c8-8d95-2877f43b404d None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:31.726287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cb3e0-d30f-40c8-8d95-2877f43b404d None admin] PATH_INFO: `/v3/projects/c2159628109c41eabb560834d45497f4/users/5812de70afbe44778a0321486e3e7ec8/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:31.728240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738cb3e0-d30f-40c8-8d95-2877f43b404d None admin] RBAC: Authorizing `identity:create_grant(project_id=c2159628109c41eabb560834d45497f4, user_id=5812de70afbe44778a0321486e3e7ec8, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:31.729669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738cb3e0-d30f-40c8-8d95-2877f43b404d None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:31.743151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 445/871] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:39:31 2022] PUT /identity/v3/projects/c2159628109c41eabb560834d45497f4/users/5812de70afbe44778a0321486e3e7ec8/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:39:33.922391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a0f4be-411c-4ef7-b0af-25581c8def11 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:33.922944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a0f4be-411c-4ef7-b0af-25581c8def11 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:33.922944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a0f4be-411c-4ef7-b0af-25581c8def11 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:33.924346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 427/872] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:33.929605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d6b4c-a922-4e05-bb23-fde4d555f5d7 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:33.930115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d6b4c-a922-4e05-bb23-fde4d555f5d7 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:33.930226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d6b4c-a922-4e05-bb23-fde4d555f5d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:33.948952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a40d6b4c-a922-4e05-bb23-fde4d555f5d7 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:33.950188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a40d6b4c-a922-4e05-bb23-fde4d555f5d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:33.959115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:34.017487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 446/873] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:34.285094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a0f4be-411c-4ef7-b0af-25581c8def11 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:34.354459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:34.386482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a0f4be-411c-4ef7-b0af-25581c8def11 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:34.419309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8aa3c8-94bb-4a8e-a94d-094d96c3b6c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:34.419919 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8aa3c8-94bb-4a8e-a94d-094d96c3b6c0 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:34.420226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8aa3c8-94bb-4a8e-a94d-094d96c3b6c0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:34.420502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8aa3c8-94bb-4a8e-a94d-094d96c3b6c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:34.430984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8aa3c8-94bb-4a8e-a94d-094d96c3b6c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-ea8aa3c8-94bb-4a8e-a94d-094d96c3b6c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:39:34.435792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:34.436839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:34.437685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 428/874] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:39:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:34.443118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d6b4c-a922-4e05-bb23-fde4d555f5d7 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:34.542565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d6b4c-a922-4e05-bb23-fde4d555f5d7 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:34.575721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:34.576350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:34.576634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:34.576909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:34.577569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:34.577880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:34.579252 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:34.587702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 447/875] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:39:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:34.593783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8aa3c8-94bb-4a8e-a94d-094d96c3b6c0 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:34.693691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8aa3c8-94bb-4a8e-a94d-094d96c3b6c0 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:34.721903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:34.727954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f33c1ca-b178-4087-a788-9fac4212cdd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:34.728648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f33c1ca-b178-4087-a788-9fac4212cdd7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:34.728953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f33c1ca-b178-4087-a788-9fac4212cdd7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:34.729230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f33c1ca-b178-4087-a788-9fac4212cdd7 None admin] PATH_INFO: `/v3/users/5812de70afbe44778a0321486e3e7ec8` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:34.730349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f33c1ca-b178-4087-a788-9fac4212cdd7 None admin] RBAC: Authorizing `identity:get_user(user_id=5812de70afbe44778a0321486e3e7ec8)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:34.731789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f33c1ca-b178-4087-a788-9fac4212cdd7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:34.733408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 429/876] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:34 2022] GET /identity/v3/users/5812de70afbe44778a0321486e3e7ec8 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:34.739374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:34.749439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:34.836632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2f076b-d427-47fb-bf53-a8bb2a0b1468 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:34.871101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3989c1f8-1e01-4ea8-b173-6a7b57251a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:34.871701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989c1f8-1e01-4ea8-b173-6a7b57251a65 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:34.871980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989c1f8-1e01-4ea8-b173-6a7b57251a65 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:34.872283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989c1f8-1e01-4ea8-b173-6a7b57251a65 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:34.938998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3989c1f8-1e01-4ea8-b173-6a7b57251a65 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:34.940474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3989c1f8-1e01-4ea8-b173-6a7b57251a65 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:34.948922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:35.043044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 448/877] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:39:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 24 08:39:35.049011 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f33c1ca-b178-4087-a788-9fac4212cdd7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:35.156365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f33c1ca-b178-4087-a788-9fac4212cdd7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:35.195485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d748eff-1d5e-4042-ad42-34337780db45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:35.196117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d748eff-1d5e-4042-ad42-34337780db45 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:35.196421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d748eff-1d5e-4042-ad42-34337780db45 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:35.196690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d748eff-1d5e-4042-ad42-34337780db45 None admin] PATH_INFO: `/v3/projects/c2159628109c41eabb560834d45497f4` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:35.198559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d748eff-1d5e-4042-ad42-34337780db45 None admin] RBAC: Authorizing `identity:get_project(project_id=c2159628109c41eabb560834d45497f4)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:35.200019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d748eff-1d5e-4042-ad42-34337780db45 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:35.201706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 430/878] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:39:35 2022] GET /identity/v3/projects/c2159628109c41eabb560834d45497f4 => generated 311 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:35.207929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3989c1f8-1e01-4ea8-b173-6a7b57251a65 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:35.304184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3989c1f8-1e01-4ea8-b173-6a7b57251a65 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:35.311470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:35.338900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7c53c-912f-43b1-89a5-0cb2e0a05494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:35.339503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7c53c-912f-43b1-89a5-0cb2e0a05494 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:35.339780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7c53c-912f-43b1-89a5-0cb2e0a05494 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:35.340050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7c53c-912f-43b1-89a5-0cb2e0a05494 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:35.340801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f7c53c-912f-43b1-89a5-0cb2e0a05494 None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=c2159628109c41eabb560834d45497f4, user.id=5812de70afbe44778a0321486e3e7ec8) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:35.341113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f7c53c-912f-43b1-89a5-0cb2e0a05494 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:35.342522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f7c53c-912f-43b1-89a5-0cb2e0a05494 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:35.348556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 449/879] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:39:35 2022] GET /identity/v3/role_assignments?user.id=5812de70afbe44778a0321486e3e7ec8&scope.project.id=c2159628109c41eabb560834d45497f4&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:37.552823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6b2603-4193-47fb-bc74-97b6f04c7c28 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:37.553398 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6b2603-4193-47fb-bc74-97b6f04c7c28 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:37.553398 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6b2603-4193-47fb-bc74-97b6f04c7c28 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:37.554717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 431/880] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:37.560915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566c943b-e0f0-420f-804d-7adc3b518b96 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:37.561592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566c943b-e0f0-420f-804d-7adc3b518b96 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:37.561822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566c943b-e0f0-420f-804d-7adc3b518b96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:37.579691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-566c943b-e0f0-420f-804d-7adc3b518b96 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:37.580891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-566c943b-e0f0-420f-804d-7adc3b518b96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:37.647841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 450/881] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:37.922688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6b2603-4193-47fb-bc74-97b6f04c7c28 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:37.992733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:38.030072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6b2603-4193-47fb-bc74-97b6f04c7c28 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:38.063872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a71462f-5652-4458-aa25-c72850df053e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:38.064494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a71462f-5652-4458-aa25-c72850df053e None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:38.064783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a71462f-5652-4458-aa25-c72850df053e None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:38.065058 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a71462f-5652-4458-aa25-c72850df053e None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:38.065721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a71462f-5652-4458-aa25-c72850df053e None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:38.067083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a71462f-5652-4458-aa25-c72850df053e None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:38.083559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 432/882] 10.222.0.39 () {60 vars in 1138 bytes} [Mon Oct 24 08:39:37 2022] POST /identity/v3/domains => generated 267 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:39:40.334985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2027bc-0e56-470d-8dac-d3a82a8d709b None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:40.335502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2027bc-0e56-470d-8dac-d3a82a8d709b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:40.335502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2027bc-0e56-470d-8dac-d3a82a8d709b None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:40.336826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 451/883] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:40.342158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd513da4-eef3-4ae4-9179-5f4c56270f51 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:40.342558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd513da4-eef3-4ae4-9179-5f4c56270f51 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:40.342716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd513da4-eef3-4ae4-9179-5f4c56270f51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:40.361141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-dd513da4-eef3-4ae4-9179-5f4c56270f51 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:40.362317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-dd513da4-eef3-4ae4-9179-5f4c56270f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:40.427292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 433/884] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:40.704621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2027bc-0e56-470d-8dac-d3a82a8d709b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:40.800775 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:40.805041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2027bc-0e56-470d-8dac-d3a82a8d709b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:40.841007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015a9bf2-fbca-4be9-a86b-bb35acf13123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:40.841632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a9bf2-fbca-4be9-a86b-bb35acf13123 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:40.841914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a9bf2-fbca-4be9-a86b-bb35acf13123 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:40.842194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a9bf2-fbca-4be9-a86b-bb35acf13123 None admin] PATH_INFO: `/v3/domains/ab764989d72d4d0b9f2d89cb1c225325` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:40.852977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015a9bf2-fbca-4be9-a86b-bb35acf13123 None admin] RBAC: Authorizing `identity:get_domain(domain_id=ab764989d72d4d0b9f2d89cb1c225325)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:40.854402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015a9bf2-fbca-4be9-a86b-bb35acf13123 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:40.856694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 452/885] 10.222.0.39 () {56 vars in 1183 bytes} [Mon Oct 24 08:39:40 2022] GET /identity/v3/domains/ab764989d72d4d0b9f2d89cb1c225325 => generated 267 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:40.862547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd513da4-eef3-4ae4-9179-5f4c56270f51 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:40.918563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:40.963112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd513da4-eef3-4ae4-9179-5f4c56270f51 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:40.998653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10e0557-2526-41ca-b30e-c806a79bf2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:40.999261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e0557-2526-41ca-b30e-c806a79bf2e1 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:40.999532 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e0557-2526-41ca-b30e-c806a79bf2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:40.999795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e0557-2526-41ca-b30e-c806a79bf2e1 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:41.000480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10e0557-2526-41ca-b30e-c806a79bf2e1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:41.001901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10e0557-2526-41ca-b30e-c806a79bf2e1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:41.028955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 434/886] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:39:40 2022] POST /identity/v3/projects => generated 361 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:39:43.244479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2f1c3-97b6-4d58-ae69-4b02bb640244 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:43.244479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2f1c3-97b6-4d58-ae69-4b02bb640244 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:43.244479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2f1c3-97b6-4d58-ae69-4b02bb640244 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:43.246240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 453/887] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:43 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:43.251893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a992b1-43e1-415c-95e4-7eb88f4b3275 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:43.252400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a992b1-43e1-415c-95e4-7eb88f4b3275 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:43.252500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a992b1-43e1-415c-95e4-7eb88f4b3275 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:43.270816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-54a992b1-43e1-415c-95e4-7eb88f4b3275 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:43.271987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-54a992b1-43e1-415c-95e4-7eb88f4b3275 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:43.339282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 435/888] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:43.611257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2f1c3-97b6-4d58-ae69-4b02bb640244 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:43.714046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2f1c3-97b6-4d58-ae69-4b02bb640244 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:43.750435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632e5751-0c49-4998-8ce0-3061617745ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:43.751068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e5751-0c49-4998-8ce0-3061617745ec None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:43.751339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e5751-0c49-4998-8ce0-3061617745ec None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:43.751609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e5751-0c49-4998-8ce0-3061617745ec None admin] PATH_INFO: `/v3/domains/ab764989d72d4d0b9f2d89cb1c225325` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:43.752754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632e5751-0c49-4998-8ce0-3061617745ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=ab764989d72d4d0b9f2d89cb1c225325)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:43.754187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632e5751-0c49-4998-8ce0-3061617745ec None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:43.755772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 454/889] 10.222.0.39 () {56 vars in 1183 bytes} [Mon Oct 24 08:39:43 2022] GET /identity/v3/domains/ab764989d72d4d0b9f2d89cb1c225325 => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:43.761938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a992b1-43e1-415c-95e4-7eb88f4b3275 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:43.801852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:43.862946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a992b1-43e1-415c-95e4-7eb88f4b3275 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:43.897490 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ae0daa-0124-4c98-9e00-6390445a7b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:43.898091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ae0daa-0124-4c98-9e00-6390445a7b9d None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:43.898369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ae0daa-0124-4c98-9e00-6390445a7b9d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:43.898642 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ae0daa-0124-4c98-9e00-6390445a7b9d None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:43.899282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ae0daa-0124-4c98-9e00-6390445a7b9d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:43.900782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ae0daa-0124-4c98-9e00-6390445a7b9d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:43.922193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 436/890] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:39:43 2022] POST /identity/v3/users => generated 311 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:39:46.175254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ba67de-4cdc-45ed-b06c-059f2e4fcf2f None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:46.175806 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ba67de-4cdc-45ed-b06c-059f2e4fcf2f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:46.175806 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ba67de-4cdc-45ed-b06c-059f2e4fcf2f None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:46.177158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 455/891] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:46.186947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacc8f20-0d1f-4727-b99d-e767f9807ef9 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:46.187440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacc8f20-0d1f-4727-b99d-e767f9807ef9 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:46.187531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacc8f20-0d1f-4727-b99d-e767f9807ef9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:46.206583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-cacc8f20-0d1f-4727-b99d-e767f9807ef9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:46.207787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cacc8f20-0d1f-4727-b99d-e767f9807ef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:46.278278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 437/892] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:39:46.548453 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ba67de-4cdc-45ed-b06c-059f2e4fcf2f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:46.577411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:46.653770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ba67de-4cdc-45ed-b06c-059f2e4fcf2f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:46.690426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4bc6f0-b1e2-49b0-a7ba-e51c5c766fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:46.691076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4bc6f0-b1e2-49b0-a7ba-e51c5c766fe5 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:46.691354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4bc6f0-b1e2-49b0-a7ba-e51c5c766fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:46.691624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4bc6f0-b1e2-49b0-a7ba-e51c5c766fe5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:46.703227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4bc6f0-b1e2-49b0-a7ba-e51c5c766fe5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-ff4bc6f0-b1e2-49b0-a7ba-e51c5c766fe5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:39:46.712042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:46.713603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:39:46.714829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 456/893] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:39:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:46.720215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacc8f20-0d1f-4727-b99d-e767f9807ef9 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:46.745846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:46.823832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacc8f20-0d1f-4727-b99d-e767f9807ef9 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:46.859934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:46.860569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:46.860861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:46.861134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:46.861788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:46.862083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:46.863440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:46.872665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 438/894] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:39:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:46.879006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4bc6f0-b1e2-49b0-a7ba-e51c5c766fe5 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:46.974568 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:46.986548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4bc6f0-b1e2-49b0-a7ba-e51c5c766fe5 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:47.023689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a91ad7-1cb4-40ae-9403-2aa2cc299a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:47.024350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a91ad7-1cb4-40ae-9403-2aa2cc299a4b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:47.024637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a91ad7-1cb4-40ae-9403-2aa2cc299a4b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:47.024968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a91ad7-1cb4-40ae-9403-2aa2cc299a4b None admin] PATH_INFO: `/v3/users/238032be7f6a4cad8cdcd48393c444a0` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:47.038627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a91ad7-1cb4-40ae-9403-2aa2cc299a4b None admin] RBAC: Authorizing `identity:get_user(user_id=238032be7f6a4cad8cdcd48393c444a0)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:47.040071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a91ad7-1cb4-40ae-9403-2aa2cc299a4b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:47.042588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 457/895] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:46 2022] GET /identity/v3/users/238032be7f6a4cad8cdcd48393c444a0 => generated 311 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:47.049019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:47.146603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:47.163852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91d1c0-7d66-4d27-9016-572a26d6d8f9 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:47.200574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b25857b-cbbd-49bd-bd5e-4081f617ac55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:47.201236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b25857b-cbbd-49bd-bd5e-4081f617ac55 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:47.201331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b25857b-cbbd-49bd-bd5e-4081f617ac55 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:47.201598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b25857b-cbbd-49bd-bd5e-4081f617ac55 None admin] PATH_INFO: `/v3/projects/b2f0d6fd01ec467bb579654551c5db30` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:47.202717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b25857b-cbbd-49bd-bd5e-4081f617ac55 None admin] RBAC: Authorizing `identity:get_project(project_id=b2f0d6fd01ec467bb579654551c5db30)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:47.204171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b25857b-cbbd-49bd-bd5e-4081f617ac55 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:47.205796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 439/896] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:39:47 2022] GET /identity/v3/projects/b2f0d6fd01ec467bb579654551c5db30 => generated 361 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:47.211802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a91ad7-1cb4-40ae-9403-2aa2cc299a4b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:47.313754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a91ad7-1cb4-40ae-9403-2aa2cc299a4b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:47.346497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5473392c-583a-458b-8014-aed8eaf2e1e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:47.347145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5473392c-583a-458b-8014-aed8eaf2e1e9 None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:47.347426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5473392c-583a-458b-8014-aed8eaf2e1e9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:47.347699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5473392c-583a-458b-8014-aed8eaf2e1e9 None admin] PATH_INFO: `/v3/projects/b2f0d6fd01ec467bb579654551c5db30/users/238032be7f6a4cad8cdcd48393c444a0/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:47.349632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5473392c-583a-458b-8014-aed8eaf2e1e9 None admin] RBAC: Authorizing `identity:create_grant(project_id=b2f0d6fd01ec467bb579654551c5db30, user_id=238032be7f6a4cad8cdcd48393c444a0, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:47.351051 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5473392c-583a-458b-8014-aed8eaf2e1e9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:47.363984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 458/897] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:39:47 2022] PUT /identity/v3/projects/b2f0d6fd01ec467bb579654551c5db30/users/238032be7f6a4cad8cdcd48393c444a0/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:39:49.670764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f98095-a1ea-4111-aa8c-0d95188107fe None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:49.671333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f98095-a1ea-4111-aa8c-0d95188107fe None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:49.671333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f98095-a1ea-4111-aa8c-0d95188107fe None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:49.672681 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 440/898] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:39:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:39:49.678294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d736ed-fde9-4822-a721-68eaba581afa None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:49.678722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d736ed-fde9-4822-a721-68eaba581afa None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:49.678832 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d736ed-fde9-4822-a721-68eaba581afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:49.691850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:49.697908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-20d736ed-fde9-4822-a721-68eaba581afa None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:39:49.699454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-20d736ed-fde9-4822-a721-68eaba581afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:39:49.774281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 459/899] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:39:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:39:50.043323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f98095-a1ea-4111-aa8c-0d95188107fe None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:50.146781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f98095-a1ea-4111-aa8c-0d95188107fe None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:50.180945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2026c4a5-080a-42bd-b9e3-84c6beb1d314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:50.182220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2026c4a5-080a-42bd-b9e3-84c6beb1d314 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:50.182501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2026c4a5-080a-42bd-b9e3-84c6beb1d314 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:50.182771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2026c4a5-080a-42bd-b9e3-84c6beb1d314 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:50.193386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2026c4a5-080a-42bd-b9e3-84c6beb1d314 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-2026c4a5-080a-42bd-b9e3-84c6beb1d314 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:39:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:39:50.200311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:39:50.201463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 441/900] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:39:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:39:50.206480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d736ed-fde9-4822-a721-68eaba581afa None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:50.305108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d736ed-fde9-4822-a721-68eaba581afa None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:50.339854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:50.340478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:50.340758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:50.341029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:50.341681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:50.341979 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:50.343337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:50.345502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:50.352057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 460/901] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:39:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:50.359203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2026c4a5-080a-42bd-b9e3-84c6beb1d314 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:50.376476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:50.461554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2026c4a5-080a-42bd-b9e3-84c6beb1d314 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:50.497110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50de305-003c-4a38-bffe-ebe4eb0f18cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:50.497735 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50de305-003c-4a38-bffe-ebe4eb0f18cb None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:50.498016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50de305-003c-4a38-bffe-ebe4eb0f18cb None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:50.498286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50de305-003c-4a38-bffe-ebe4eb0f18cb None admin] PATH_INFO: `/v3/users/238032be7f6a4cad8cdcd48393c444a0` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:50.499430 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50de305-003c-4a38-bffe-ebe4eb0f18cb None admin] RBAC: Authorizing `identity:get_user(user_id=238032be7f6a4cad8cdcd48393c444a0)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:50.500877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50de305-003c-4a38-bffe-ebe4eb0f18cb None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:50.502449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 442/902] 10.222.0.39 () {56 vars in 1177 bytes} [Mon Oct 24 08:39:50 2022] GET /identity/v3/users/238032be7f6a4cad8cdcd48393c444a0 => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:39:50.508996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:50.619990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd512e7d-45c4-4a70-9d0a-6fb6462f8de4 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:50.654091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba83570-2849-4304-92a4-7daaeab73b8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:50.654694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba83570-2849-4304-92a4-7daaeab73b8b None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:50.654971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba83570-2849-4304-92a4-7daaeab73b8b None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:50.655241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba83570-2849-4304-92a4-7daaeab73b8b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:50.718055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:50.721297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba83570-2849-4304-92a4-7daaeab73b8b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:50.723257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba83570-2849-4304-92a4-7daaeab73b8b None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:50.825255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 461/903] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:39:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 24 08:39:50.831801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50de305-003c-4a38-bffe-ebe4eb0f18cb None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:50.915617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:50.933776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50de305-003c-4a38-bffe-ebe4eb0f18cb None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:50.968917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa62d64b-6ea1-45a5-b7c9-01cc881829e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:50.969543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa62d64b-6ea1-45a5-b7c9-01cc881829e8 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:50.969829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa62d64b-6ea1-45a5-b7c9-01cc881829e8 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:50.970100 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa62d64b-6ea1-45a5-b7c9-01cc881829e8 None admin] PATH_INFO: `/v3/projects/b2f0d6fd01ec467bb579654551c5db30` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:50.971221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa62d64b-6ea1-45a5-b7c9-01cc881829e8 None admin] RBAC: Authorizing `identity:get_project(project_id=b2f0d6fd01ec467bb579654551c5db30)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:50.972765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa62d64b-6ea1-45a5-b7c9-01cc881829e8 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:50.974421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 443/904] 10.222.0.39 () {56 vars in 1186 bytes} [Mon Oct 24 08:39:50 2022] GET /identity/v3/projects/b2f0d6fd01ec467bb579654551c5db30 => generated 361 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:39:50.980930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba83570-2849-4304-92a4-7daaeab73b8b None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:39:51.084803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:39:51.089857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba83570-2849-4304-92a4-7daaeab73b8b None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:39:51.124644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157649c3-c16d-4e18-9ce0-815582356f82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:39:51.125243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157649c3-c16d-4e18-9ce0-815582356f82 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:39:51.125525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157649c3-c16d-4e18-9ce0-815582356f82 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:39:51.125797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157649c3-c16d-4e18-9ce0-815582356f82 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:39:51.126516 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157649c3-c16d-4e18-9ce0-815582356f82 None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=b2f0d6fd01ec467bb579654551c5db30, user.id=238032be7f6a4cad8cdcd48393c444a0) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:39:51.126825 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157649c3-c16d-4e18-9ce0-815582356f82 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:39:51.128266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157649c3-c16d-4e18-9ce0-815582356f82 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:39:51.134769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 462/905] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:39:50 2022] GET /identity/v3/role_assignments?user.id=238032be7f6a4cad8cdcd48393c444a0&scope.project.id=b2f0d6fd01ec467bb579654551c5db30&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:40:00.964637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=25 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:01.131618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=19 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:49.700976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff119203-32bc-4e3f-9265-214a89c69be9 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:49.701888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff119203-32bc-4e3f-9265-214a89c69be9 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:49.701888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff119203-32bc-4e3f-9265-214a89c69be9 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:49.709586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 444/906] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:40:49 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:40:49.713291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe23b27-eb7c-494e-b687-d358909bb3b8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:49.713808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe23b27-eb7c-494e-b687-d358909bb3b8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:49.713905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe23b27-eb7c-494e-b687-d358909bb3b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:49.736799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-bbe23b27-eb7c-494e-b687-d358909bb3b8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:40:49.738414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bbe23b27-eb7c-494e-b687-d358909bb3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:40:49.852641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 463/907] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:40:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:40:50.157491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff119203-32bc-4e3f-9265-214a89c69be9 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:50.460197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff119203-32bc-4e3f-9265-214a89c69be9 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:50.509060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:50.509927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:50.510236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:50.510571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:50.525042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:50.527096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:40:50.560320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:40:50.561586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:40:50.562601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:40:50.562601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:40:50.562601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:40:50.567163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 445/908] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:40:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 409 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:40:50.568713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe23b27-eb7c-494e-b687-d358909bb3b8 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:50.708222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe23b27-eb7c-494e-b687-d358909bb3b8 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:50.764109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:50.775375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72f3d01-a9af-4805-b2b6-bf6578b75c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:50.776007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f3d01-a9af-4805-b2b6-bf6578b75c01 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:50.776319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f3d01-a9af-4805-b2b6-bf6578b75c01 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:50.776591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f3d01-a9af-4805-b2b6-bf6578b75c01 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:50.777345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72f3d01-a9af-4805-b2b6-bf6578b75c01 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:40:50.777611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72f3d01-a9af-4805-b2b6-bf6578b75c01 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:50.778926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72f3d01-a9af-4805-b2b6-bf6578b75c01 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:50.792849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 464/909] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:40:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:40:50.799423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:50.891737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:50.960499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79aadb8-540f-46fe-9935-37ff75b41ae4 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:51.018194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0eccbb9-e7fa-496b-b067-2cbedce56a46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:51.018724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eccbb9-e7fa-496b-b067-2cbedce56a46 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:51.019054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eccbb9-e7fa-496b-b067-2cbedce56a46 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:51.019260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eccbb9-e7fa-496b-b067-2cbedce56a46 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:51.019916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0eccbb9-e7fa-496b-b067-2cbedce56a46 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:51.021414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0eccbb9-e7fa-496b-b067-2cbedce56a46 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:51.055907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 446/910] 10.222.0.39 () {60 vars in 1132 bytes} [Mon Oct 24 08:40:50 2022] POST /identity/v3/users => generated 251 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:40:53.558473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e752c44-4359-424d-bc83-6c70e492f71f None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:53.559090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e752c44-4359-424d-bc83-6c70e492f71f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:53.559460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e752c44-4359-424d-bc83-6c70e492f71f None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:53.564388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 465/911] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:40:53 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:40:53.569318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b1c52-701d-4ab1-867c-8078648ab46b None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:53.569318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b1c52-701d-4ab1-867c-8078648ab46b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:53.569318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b1c52-701d-4ab1-867c-8078648ab46b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:53.592861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-380b1c52-701d-4ab1-867c-8078648ab46b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:40:53.594188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-380b1c52-701d-4ab1-867c-8078648ab46b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:40:53.696786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 447/912] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:40:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 24 08:40:53.980218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e752c44-4359-424d-bc83-6c70e492f71f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:54.147895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e752c44-4359-424d-bc83-6c70e492f71f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:54.201640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cec67a-4263-4a6f-ad82-55c0f01de0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:54.202511 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cec67a-4263-4a6f-ad82-55c0f01de0c8 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:54.203061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cec67a-4263-4a6f-ad82-55c0f01de0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:54.203563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cec67a-4263-4a6f-ad82-55c0f01de0c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:54.223907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cec67a-4263-4a6f-ad82-55c0f01de0c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-b4cec67a-4263-4a6f-ad82-55c0f01de0c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:40:54.231383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:40:54.233304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:40:54.234572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 466/913] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:40:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:40:54.240165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380b1c52-701d-4ab1-867c-8078648ab46b None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:54.300089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:54.408257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380b1c52-701d-4ab1-867c-8078648ab46b None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:54.470371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:54.471229 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:54.471794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:54.472275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:54.473483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] RBAC: Adding query filter params (name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:40:54.474060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:54.476358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:54.496274 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 448/914] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:40:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:40:54.498354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cec67a-4263-4a6f-ad82-55c0f01de0c8 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:54.571657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:54.655634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cec67a-4263-4a6f-ad82-55c0f01de0c8 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:54.702681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:54.703332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:54.703634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:54.703928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:54.717919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:54.719428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:40:54.740177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:40:54.741679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:40:54.743416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:40:54.743416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:40:54.743416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:40:54.743416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 467/915] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:40:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:40:54.750890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:54.911087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:54.916733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ce8a3-c671-4873-b35f-dc2aa3443f2c None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:54.978555 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:54.981076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:54.981076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:54.981076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:54.983094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:40:54.983094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:54.985269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:55.012126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 449/916] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:40:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:40:55.019623 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:55.184090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0225ef57-ab31-40ff-ab4f-ed4388957143 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:55.187084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:55.262040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:55.263212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:55.263632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:55.264121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] PATH_INFO: `/v3/users/placement` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:55.279685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:55.282763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:40:55.306093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:40:55.307993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:40:55.318007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 468/917] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:40:55 2022] GET /identity/v3/users/placement => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:40:55.320470 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:55.502022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0276ee01-7ac4-4116-9fc5-4286cce18542 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:55.551183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:55.551672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:55.552003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:55.552328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:55.553504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:40:55.553828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:55.556261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:55.587960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 450/918] 10.222.0.39 () {56 vars in 1143 bytes} [Mon Oct 24 08:40:55 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:40:55.596085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:55.788298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf6b0a-2bf4-4ecc-a5c7-0dbcb5cfebec None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:55.850801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:55.851775 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:55.852372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:55.852853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:55.873757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:55.876573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:40:55.906441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:40:55.907569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:40:55.908620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:40:55.908620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:40:55.908620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:40:55.912447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 469/919] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:40:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:40:55.917357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:55.951096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:56.117606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0e7243-1d56-4550-be21-70075d7dcf44 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:56.171235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:56.172358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:56.172844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:56.173305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:56.174506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:40:56.175031 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:56.177724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:56.204065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 451/920] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:40:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:40:56.205145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:56.228319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:56.374184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2552adc2-b472-417b-9574-bf930bd03def None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:56.429879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:56.431473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:56.431951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:56.432482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:56.444656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:56.447314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:40:56.470454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:40:56.472380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:40:56.474122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 470/921] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:40:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:40:56.480722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:56.616384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:56.653934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34aae731-a79c-4389-ae7a-355a613a5fa9 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:56.708642 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d8867a-a77e-4f23-baaa-4e921bd6a086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:56.709362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8867a-a77e-4f23-baaa-4e921bd6a086 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:56.709745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8867a-a77e-4f23-baaa-4e921bd6a086 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:56.710122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8867a-a77e-4f23-baaa-4e921bd6a086 None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:56.711534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8867a-a77e-4f23-baaa-4e921bd6a086 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:40:56.711723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8867a-a77e-4f23-baaa-4e921bd6a086 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:56.713558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8867a-a77e-4f23-baaa-4e921bd6a086 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:56.728760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 452/922] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:40:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:40:56.737690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:40:56.844185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:40:56.885692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2109b1db-2310-4044-8c91-df2565237805 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:40:56.943106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3400fa9b-4845-4ba4-bb3f-ecc15d195dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:40:56.943694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3400fa9b-4845-4ba4-bb3f-ecc15d195dfd None admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:56.944079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3400fa9b-4845-4ba4-bb3f-ecc15d195dfd None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:56.944519 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3400fa9b-4845-4ba4-bb3f-ecc15d195dfd None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/0c28fe744b614210a21aca0a7395089f/roles/a1265c18dedb42d5943905f144e48fb7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:56.966779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3400fa9b-4845-4ba4-bb3f-ecc15d195dfd None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=0c28fe744b614210a21aca0a7395089f, role_id=a1265c18dedb42d5943905f144e48fb7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:40:56.968714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3400fa9b-4845-4ba4-bb3f-ecc15d195dfd None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:40:56.991245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 471/923] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:40:56 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/0c28fe744b614210a21aca0a7395089f/roles/a1265c18dedb42d5943905f144e48fb7 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:40:59.588705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c526866-abc0-4185-908a-ff624c20639f None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:59.589316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c526866-abc0-4185-908a-ff624c20639f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:59.589316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c526866-abc0-4185-908a-ff624c20639f None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:59.590755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 453/924] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:40:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:40:59.596671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d32dbd-f3b6-40b9-9a54-c91a53014ee6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:40:59.597197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d32dbd-f3b6-40b9-9a54-c91a53014ee6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:40:59.597197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d32dbd-f3b6-40b9-9a54-c91a53014ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:40:59.615427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c4d32dbd-f3b6-40b9-9a54-c91a53014ee6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:40:59.616723 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c4d32dbd-f3b6-40b9-9a54-c91a53014ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:40:59.707467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 472/925] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:40:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:41:00.010094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c526866-abc0-4185-908a-ff624c20639f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:00.162271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c526866-abc0-4185-908a-ff624c20639f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:00.206192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07008034-5f29-4a55-b326-cfb6d5b1bc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:00.207219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07008034-5f29-4a55-b326-cfb6d5b1bc4d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:00.207568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07008034-5f29-4a55-b326-cfb6d5b1bc4d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:00.208041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07008034-5f29-4a55-b326-cfb6d5b1bc4d None admin] PATH_INFO: `/v3/roles/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:00.219461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:00.226045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07008034-5f29-4a55-b326-cfb6d5b1bc4d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-07008034-5f29-4a55-b326-cfb6d5b1bc4d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:41:00.232552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:00.234670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:00.235596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 454/926] 10.222.0.39 () {56 vars in 1102 bytes} [Mon Oct 24 08:41:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:00.245651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d32dbd-f3b6-40b9-9a54-c91a53014ee6 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:00.323052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:00.385558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d32dbd-f3b6-40b9-9a54-c91a53014ee6 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:00.427933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:00.428697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:00.428970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:00.429232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:00.430247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] RBAC: Adding query filter params (name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:00.431084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:00.432510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:00.447660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 473/927] 10.222.0.39 () {56 vars in 1103 bytes} [Mon Oct 24 08:41:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:41:00.449656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07008034-5f29-4a55-b326-cfb6d5b1bc4d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:00.583763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07008034-5f29-4a55-b326-cfb6d5b1bc4d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:00.635631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:00.636540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:00.636898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:00.637360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:00.652009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:00.653500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:41:00.671358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:00.672575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:41:00.674188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:41:00.674188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:00.674188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:00.674188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 455/928] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:00.680555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:00.884976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d60bbba-a24b-43cf-9a02-f44e00f0fed6 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:00.912199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:00.943938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:00.946648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:00.946648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:00.946957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:00.947764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:00.948089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:00.950106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:00.975143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 474/929] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:41:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:00.984024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:01.054329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:01.129289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa3c226-bdd7-4305-a668-90f1a0fb78e7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:01.175552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:01.176517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:01.176974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:01.177306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] PATH_INFO: `/v3/users/placement` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:01.188090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:01.189567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:41:01.203962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:01.206049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:01.208349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 456/930] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:00 2022] GET /identity/v3/users/placement => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:01.213779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:01.358849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f55863-f788-45a8-805e-c263c228aa9f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:01.407256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:01.407866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:01.408192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:01.408489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:01.409376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:01.409747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:01.411442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:01.435738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 475/931] 10.222.0.39 () {56 vars in 1143 bytes} [Mon Oct 24 08:41:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:01.443205 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:01.587537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abdfe45-b842-4b8f-80a6-7d462b044e59 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:01.602080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:01.637698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ba6245-1529-4693-992d-4f3b26716fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:01.638393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ba6245-1529-4693-992d-4f3b26716fc1 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:01.638617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ba6245-1529-4693-992d-4f3b26716fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:01.638994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ba6245-1529-4693-992d-4f3b26716fc1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:01.734731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ba6245-1529-4693-992d-4f3b26716fc1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:01.736170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ba6245-1529-4693-992d-4f3b26716fc1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:01.875959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 457/932] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:41:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 24 08:41:01.882877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:01.935111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:02.023703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3937a0ca-db7f-43d0-9f19-01032948a008 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:02.067802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:02.068490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:02.068749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:02.069014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:02.086224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:02.087941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:41:02.107971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:02.114997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:41:02.116346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:41:02.116346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:02.116346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:02.116346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 476/933] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:02.116758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ba6245-1529-4693-992d-4f3b26716fc1 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:02.131423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:02.232671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ba6245-1529-4693-992d-4f3b26716fc1 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:02.276605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:02.277109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:02.277387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:02.277742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:02.278483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:02.278822 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:02.282462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:02.296936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 458/934] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:41:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:02.304531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:02.464535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:02.491943 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530fbae3-5842-4d0c-8462-4df3b82826cb None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:02.543195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:02.544181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:02.544697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:02.545191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] PATH_INFO: `/v3/projects/service` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:02.558476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:02.561359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:41:02.579485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:02.580964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:02.582292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 477/935] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:41:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:02.589852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:02.683764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:02.735182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc29be30-8c40-4c41-a168-61b9f7e9600a None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:02.782314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a0e9f9-9933-4046-904e-c3392cd622fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:02.782930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0e9f9-9933-4046-904e-c3392cd622fd None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:02.783224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0e9f9-9933-4046-904e-c3392cd622fd None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:02.783485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0e9f9-9933-4046-904e-c3392cd622fd None admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:02.784215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a0e9f9-9933-4046-904e-c3392cd622fd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:02.784533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a0e9f9-9933-4046-904e-c3392cd622fd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:02.786003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a0e9f9-9933-4046-904e-c3392cd622fd None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:02.799404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 459/936] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:41:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:41:02.806456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:02.950932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e3dd2e-f652-4b2e-b531-a3638a40d04e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:03.001804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90552e77-8f0a-4f86-8987-e7323923b8e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:03.002529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90552e77-8f0a-4f86-8987-e7323923b8e9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:03.002766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90552e77-8f0a-4f86-8987-e7323923b8e9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:03.003085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90552e77-8f0a-4f86-8987-e7323923b8e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:03.003792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90552e77-8f0a-4f86-8987-e7323923b8e9 None admin] RBAC: Adding query filter params (role.id=a1265c18dedb42d5943905f144e48fb7, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=0c28fe744b614210a21aca0a7395089f) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:03.004314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90552e77-8f0a-4f86-8987-e7323923b8e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:03.005586 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90552e77-8f0a-4f86-8987-e7323923b8e9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:03.013698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 478/937] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:41:02 2022] GET /identity/v3/role_assignments?user.id=0c28fe744b614210a21aca0a7395089f&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=a1265c18dedb42d5943905f144e48fb7 => generated 608 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:05.412188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23daa1-0871-4f32-98e2-e70927271349 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:05.412730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23daa1-0871-4f32-98e2-e70927271349 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:05.412730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23daa1-0871-4f32-98e2-e70927271349 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:05.414917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 460/938] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:41:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:41:05.420160 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02cc4be-a313-47da-8b22-263b45ff2569 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:05.420722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02cc4be-a313-47da-8b22-263b45ff2569 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:05.420722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02cc4be-a313-47da-8b22-263b45ff2569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:05.444107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d02cc4be-a313-47da-8b22-263b45ff2569 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:41:05.445748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d02cc4be-a313-47da-8b22-263b45ff2569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:41:05.449988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:05.535215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 479/939] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:41:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:41:05.810172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d23daa1-0871-4f32-98e2-e70927271349 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:05.979891 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d23daa1-0871-4f32-98e2-e70927271349 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:06.026306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:06.036055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf4999-718e-4ea1-aad5-5938570849d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:06.036861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf4999-718e-4ea1-aad5-5938570849d5 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:06.037177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf4999-718e-4ea1-aad5-5938570849d5 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:06.037412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf4999-718e-4ea1-aad5-5938570849d5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:06.050275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bf4999-718e-4ea1-aad5-5938570849d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-20bf4999-718e-4ea1-aad5-5938570849d5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:41:06.056187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:06.058044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 461/940] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:41:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:06.067349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02cc4be-a313-47da-8b22-263b45ff2569 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:06.219877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02cc4be-a313-47da-8b22-263b45ff2569 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:06.260253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:06.260872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:06.261175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:06.261418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:06.262090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] RBAC: Adding query filter params (name=admin) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:06.262388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:06.263765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:06.274647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 480/941] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:41:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:06.281982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf4999-718e-4ea1-aad5-5938570849d5 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:06.412846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf4999-718e-4ea1-aad5-5938570849d5 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:06.462982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:06.463831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:06.464152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:06.464577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:06.476056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:06.478486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:41:06.500208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:06.501963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:41:06.503261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:41:06.503261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:06.503261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:06.509013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 462/942] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:06.510131 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:06.513792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:06.646878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1545bb2c-09b8-4302-8afd-f86c4bf7670f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:06.698474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:06.698976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:06.699253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:06.699521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:06.700201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:06.700508 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:06.701875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:06.717732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 481/943] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:41:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:41:06.724098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:06.792719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:06.876947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4270975-4c64-4c34-b923-3b600f495c1d None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:06.923508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:06.924175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:06.924452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:06.924724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] PATH_INFO: `/v3/users/placement` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:06.934069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:06.935692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:41:06.949262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:06.950671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:06.951729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 463/944] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:06 2022] GET /identity/v3/users/placement => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:06.956549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:07.053026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:07.106468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60403cc5-4417-497b-ab1d-054a7a709f85 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:07.154613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:07.155622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:07.156065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:07.156566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:07.157639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:07.158143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:07.160279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:07.185933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 482/945] 10.222.0.39 () {56 vars in 1143 bytes} [Mon Oct 24 08:41:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:41:07.191970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:07.321128 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ece1fc-1d1c-44c2-9102-1c1b5cbaa9c7 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:07.322975 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:07.366487 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:07.367221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:07.367591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:07.367864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:07.382134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:07.384447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:41:07.401947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:07.403930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:41:07.405851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:41:07.405851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:07.405851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:07.405851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 464/946] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:07.409986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:07.551577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bed636-6d60-4ac5-8bfd-ca04fd531dc7 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:07.581261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:07.599870 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:07.600564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:07.600840 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:07.601167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:07.602429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:07.602730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:07.604430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:07.619851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 483/947] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:41:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:07.626754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:07.752595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc85ba47-ad09-474e-ae90-b531881d11e1 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:07.794808 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:07.795577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:07.795857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:07.796268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:07.804309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:07.806428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:41:07.820613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:07.822685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:07.823776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 465/948] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:41:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:07.833452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:07.967554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba032703-ddc3-47e4-90ab-c227444dd08e None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:08.012547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad25f1b-730d-44b2-8651-792b8f5ac3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:08.013552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad25f1b-730d-44b2-8651-792b8f5ac3ae None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:08.013839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad25f1b-730d-44b2-8651-792b8f5ac3ae None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:08.014251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad25f1b-730d-44b2-8651-792b8f5ac3ae None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:08.015092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad25f1b-730d-44b2-8651-792b8f5ac3ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:08.015404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad25f1b-730d-44b2-8651-792b8f5ac3ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:08.016889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad25f1b-730d-44b2-8651-792b8f5ac3ae None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:08.033707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 484/949] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:41:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:08.037406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:08.058782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:08.173305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb1038a-d7b0-49f1-9a13-2643f0e13a84 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:08.218423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd76778-35c7-4615-92fb-f07c4fadce52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:08.219420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd76778-35c7-4615-92fb-f07c4fadce52 None admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:08.219931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd76778-35c7-4615-92fb-f07c4fadce52 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:08.220405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd76778-35c7-4615-92fb-f07c4fadce52 None admin] PATH_INFO: `/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/0c28fe744b614210a21aca0a7395089f/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:08.222931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd76778-35c7-4615-92fb-f07c4fadce52 None admin] RBAC: Authorizing `identity:create_grant(project_id=1356ba882d344f3d9e3b5d0806720c8f, user_id=0c28fe744b614210a21aca0a7395089f, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:08.224771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd76778-35c7-4615-92fb-f07c4fadce52 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:08.240630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 466/950] 10.222.0.39 () {58 vars in 1439 bytes} [Mon Oct 24 08:41:08 2022] PUT /identity/v3/projects/1356ba882d344f3d9e3b5d0806720c8f/users/0c28fe744b614210a21aca0a7395089f/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:41:10.647802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7dc6ab-ad89-4a62-9606-833ab126439e None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:10.649278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7dc6ab-ad89-4a62-9606-833ab126439e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:10.649563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7dc6ab-ad89-4a62-9606-833ab126439e None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:10.652339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 485/951] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:41:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:41:10.660067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c1ec-ceca-4142-a705-bd9d7b0872b2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:10.660969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c1ec-ceca-4142-a705-bd9d7b0872b2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:10.660969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c1ec-ceca-4142-a705-bd9d7b0872b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:10.685329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-aea7c1ec-ceca-4142-a705-bd9d7b0872b2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:41:10.687342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-aea7c1ec-ceca-4142-a705-bd9d7b0872b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:41:10.801083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 467/952] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:41:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:41:11.116289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7dc6ab-ad89-4a62-9606-833ab126439e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:11.170527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:11.260852 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7dc6ab-ad89-4a62-9606-833ab126439e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:11.309441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64a81d0-f12f-4cf8-9f9f-95646f678b0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:11.310675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64a81d0-f12f-4cf8-9f9f-95646f678b0f None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:11.311319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64a81d0-f12f-4cf8-9f9f-95646f678b0f None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:11.311851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64a81d0-f12f-4cf8-9f9f-95646f678b0f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:11.327420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64a81d0-f12f-4cf8-9f9f-95646f678b0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-c64a81d0-f12f-4cf8-9f9f-95646f678b0f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 24 08:41:11.336615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:11.338791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 24 08:41:11.340903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:11.347434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 486/953] 10.222.0.39 () {56 vars in 1096 bytes} [Mon Oct 24 08:41:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:11.349859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7c1ec-ceca-4142-a705-bd9d7b0872b2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:11.357260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:11.495871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7c1ec-ceca-4142-a705-bd9d7b0872b2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:11.548554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:11.549496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:11.549997 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:11.550465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:11.551731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:11.552443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:11.555524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:11.571727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 468/954] 10.222.0.39 () {56 vars in 1099 bytes} [Mon Oct 24 08:41:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:11.578638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64a81d0-f12f-4cf8-9f9f-95646f678b0f None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:11.725087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:11.729271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64a81d0-f12f-4cf8-9f9f-95646f678b0f None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:11.771269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:11.772005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:11.772289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:11.772694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:11.784767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:11.786308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:41:11.805865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:41:11.809302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:41:11.809302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:11.809302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:11.809302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 487/955] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:11.813530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:11.893707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:11.934000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7526c34-a0ad-4e2e-8705-1591ba852af0 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:11.984347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:11.984987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:11.985330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:11.985608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:11.986250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:11.987316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:11.989110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:12.011360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 469/956] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:41:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:12.012744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:12.136336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dd8705-0494-40f8-96dc-be656ab0f884 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:12.185007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:12.185826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:12.186261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:12.186633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] PATH_INFO: `/v3/users/placement` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:12.199644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:12.201662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 24 08:41:12.216498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:12.217817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:12.219325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 488/957] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:12 2022] GET /identity/v3/users/placement => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:12.230654 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:12.374237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d1fe0-7ee8-4276-80ad-746a33942d8b None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:12.410574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:12.424775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:12.425459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:12.425974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:12.426631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:12.427813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:12.428442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:12.431363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:12.455941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 470/958] 10.222.0.39 () {56 vars in 1143 bytes} [Mon Oct 24 08:41:12 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:41:12.466062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:12.484408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:12.606321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7ff1c-2ff3-4c78-94a5-e44f4abe0d65 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:12.648083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc1a7c-1b4f-4883-aa3e-644cb8e30efd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:12.649260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc1a7c-1b4f-4883-aa3e-644cb8e30efd None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:12.649532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc1a7c-1b4f-4883-aa3e-644cb8e30efd None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:12.649804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc1a7c-1b4f-4883-aa3e-644cb8e30efd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:12.729441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bc1a7c-1b4f-4883-aa3e-644cb8e30efd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:12.731277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bc1a7c-1b4f-4883-aa3e-644cb8e30efd None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:12.752013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:12.885432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 489/959] 10.222.0.39 () {58 vars in 1282 bytes} [Mon Oct 24 08:41:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 24 08:41:12.893913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:13.026457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c747d47-af4e-4aac-92ee-d2ae3ea39a09 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:13.078355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:13.078996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:13.079271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:13.079537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:13.091508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:13.093225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 24 08:41:13.112531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:13.114510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 24 08:41:13.115887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 24 08:41:13.115887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 24 08:41:13.115887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:13.115887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 471/960] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:41:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:13.120606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc1a7c-1b4f-4883-aa3e-644cb8e30efd None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:13.238045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc1a7c-1b4f-4883-aa3e-644cb8e30efd None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:13.255809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:13.292967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:13.293952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:13.294440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:13.294983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:13.296610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:13.299130 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:13.321242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 490/961] 10.222.0.39 () {56 vars in 1109 bytes} [Mon Oct 24 08:41:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:13.330133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:13.376727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:13.475554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9088dfb3-1946-404f-b36a-18b633b8ed98 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:13.524575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:13.525109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:13.525404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:13.525644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] PATH_INFO: `/v3/projects/service` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:13.533452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:13.534946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 24 08:41:13.548979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:13.551062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: ERROR keystone.server.flask.application Oct 24 08:41:13.553946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 472/962] 10.222.0.39 () {56 vars in 1111 bytes} [Mon Oct 24 08:41:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 24 08:41:13.557773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:13.712613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90783ab2-2c55-497f-aa57-76ae902feaf3 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:13.755718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a59fadd-3bfe-472c-a7ff-73e5f21469d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:13.756538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59fadd-3bfe-472c-a7ff-73e5f21469d6 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:13.756700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59fadd-3bfe-472c-a7ff-73e5f21469d6 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:13.756908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59fadd-3bfe-472c-a7ff-73e5f21469d6 None admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:13.757643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a59fadd-3bfe-472c-a7ff-73e5f21469d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:13.757960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a59fadd-3bfe-472c-a7ff-73e5f21469d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:13.759686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a59fadd-3bfe-472c-a7ff-73e5f21469d6 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:13.775617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 491/963] 10.222.0.39 () {56 vars in 1148 bytes} [Mon Oct 24 08:41:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:13.782012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:13.921545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:13.928926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7d97ce-ecff-456a-829e-38f3d0e63a40 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:13.991183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22230f10-9599-4353-ba11-1e85b21ecc6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:13.992958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22230f10-9599-4353-ba11-1e85b21ecc6f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:13.993649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22230f10-9599-4353-ba11-1e85b21ecc6f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:13.993905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22230f10-9599-4353-ba11-1e85b21ecc6f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:13.995076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22230f10-9599-4353-ba11-1e85b21ecc6f None admin] RBAC: Adding query filter params (role.id=73b89f24867f4f45937bb4a67856e0ba, scope.project.id=1356ba882d344f3d9e3b5d0806720c8f, user.id=0c28fe744b614210a21aca0a7395089f) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:13.995552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22230f10-9599-4353-ba11-1e85b21ecc6f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:13.998611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22230f10-9599-4353-ba11-1e85b21ecc6f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:14.010078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 473/964] 10.222.0.39 () {56 vars in 1374 bytes} [Mon Oct 24 08:41:13 2022] GET /identity/v3/role_assignments?user.id=0c28fe744b614210a21aca0a7395089f&scope.project.id=1356ba882d344f3d9e3b5d0806720c8f&role.id=73b89f24867f4f45937bb4a67856e0ba => generated 608 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:16.719403 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d74ee9-4286-448c-9387-1d22bb7bb5c2 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:16.719966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d74ee9-4286-448c-9387-1d22bb7bb5c2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:16.719966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d74ee9-4286-448c-9387-1d22bb7bb5c2 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:16.721319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 492/965] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:41:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:41:16.726638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828c86f9-17e6-4830-b78a-4008e50fcdc1 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:16.727126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828c86f9-17e6-4830-b78a-4008e50fcdc1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:16.727232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828c86f9-17e6-4830-b78a-4008e50fcdc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:16.745558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-828c86f9-17e6-4830-b78a-4008e50fcdc1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:41:16.747016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-828c86f9-17e6-4830-b78a-4008e50fcdc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:41:16.813155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 474/966] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:41:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:41:17.112056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d74ee9-4286-448c-9387-1d22bb7bb5c2 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:17.144195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:17.215886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d74ee9-4286-448c-9387-1d22bb7bb5c2 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:17.249708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:17.250298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:17.250523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:17.250792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] PATH_INFO: `/v3/services/placement` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:17.251360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:17.252738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:17.262030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:17.263966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:17.266177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 493/967] 10.222.0.39 () {56 vars in 1117 bytes} [Mon Oct 24 08:41:17 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:17.271232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828c86f9-17e6-4830-b78a-4008e50fcdc1 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:17.360278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:17.369951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828c86f9-17e6-4830-b78a-4008e50fcdc1 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:17.405224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d24deb-3b08-47c6-9868-6caad4f05b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:17.405817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d24deb-3b08-47c6-9868-6caad4f05b94 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:17.406091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d24deb-3b08-47c6-9868-6caad4f05b94 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:17.406357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d24deb-3b08-47c6-9868-6caad4f05b94 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:17.406971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d24deb-3b08-47c6-9868-6caad4f05b94 None admin] RBAC: Adding query filter params (name=placement) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:17.407266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d24deb-3b08-47c6-9868-6caad4f05b94 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:17.408652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d24deb-3b08-47c6-9868-6caad4f05b94 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:17.413333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 475/968] 10.222.0.39 () {56 vars in 1116 bytes} [Mon Oct 24 08:41:17 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:17.419694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:17.522651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:17.529827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46342ab-ab66-4e81-a785-dfb88d852205 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:17.564579 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624b43ad-5276-4a7a-b59d-fad30786adcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:17.565178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b43ad-5276-4a7a-b59d-fad30786adcc None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:17.565456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b43ad-5276-4a7a-b59d-fad30786adcc None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:17.565728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b43ad-5276-4a7a-b59d-fad30786adcc None admin] PATH_INFO: `/v3/services` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:17.566357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624b43ad-5276-4a7a-b59d-fad30786adcc None admin] RBAC: Adding query filter params (type=placement) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:17.566658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624b43ad-5276-4a7a-b59d-fad30786adcc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:17.568009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624b43ad-5276-4a7a-b59d-fad30786adcc None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:17.572542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 494/969] 10.222.0.39 () {56 vars in 1116 bytes} [Mon Oct 24 08:41:17 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:19.778640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ce0257-6408-40f6-a971-b06ffd97c0f0 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:19.779172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ce0257-6408-40f6-a971-b06ffd97c0f0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:19.779172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ce0257-6408-40f6-a971-b06ffd97c0f0 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:19.780554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 476/970] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:41:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:41:19.785842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0e63ff-8c62-473a-bdfa-347b99c1f677 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:19.786261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0e63ff-8c62-473a-bdfa-347b99c1f677 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:19.786370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0e63ff-8c62-473a-bdfa-347b99c1f677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:19.805107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-dc0e63ff-8c62-473a-bdfa-347b99c1f677 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:41:19.806454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-dc0e63ff-8c62-473a-bdfa-347b99c1f677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:41:19.878973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 495/971] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:41:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:41:20.153880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ce0257-6408-40f6-a971-b06ffd97c0f0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:20.260906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ce0257-6408-40f6-a971-b06ffd97c0f0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:20.296821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c23d42-8513-411d-8b4b-30dd35e96577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:20.297431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c23d42-8513-411d-8b4b-30dd35e96577 None admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:20.297714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c23d42-8513-411d-8b4b-30dd35e96577 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:20.297988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c23d42-8513-411d-8b4b-30dd35e96577 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:20.298657 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c23d42-8513-411d-8b4b-30dd35e96577 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:20.300022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c23d42-8513-411d-8b4b-30dd35e96577 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:20.315636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 477/972] 10.222.0.39 () {60 vars in 1141 bytes} [Mon Oct 24 08:41:20 2022] POST /identity/v3/services => generated 246 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:41:22.555667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ef68dd-9450-485d-9096-e9609e959b0a None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:22.556305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ef68dd-9450-485d-9096-e9609e959b0a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:22.556305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ef68dd-9450-485d-9096-e9609e959b0a None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:22.557761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 496/973] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:41:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:41:22.563460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b92ec88-ce6e-4a93-ac3a-7bc626016a76 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:22.563920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b92ec88-ce6e-4a93-ac3a-7bc626016a76 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:22.564079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b92ec88-ce6e-4a93-ac3a-7bc626016a76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:22.583047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7b92ec88-ce6e-4a93-ac3a-7bc626016a76 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:41:22.584294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7b92ec88-ce6e-4a93-ac3a-7bc626016a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:41:22.594847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:22.661671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 478/974] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:41:22 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:41:22.933848 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ef68dd-9450-485d-9096-e9609e959b0a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:23.057574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:23.069363 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ef68dd-9450-485d-9096-e9609e959b0a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:23.104486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:23.105106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:23.105380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:23.105660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] PATH_INFO: `/v3/services/placement` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:23.106224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:23.107580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:23.116441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:23.118265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:23.119351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 497/975] 10.222.0.39 () {56 vars in 1117 bytes} [Mon Oct 24 08:41:22 2022] GET /identity/v3/services/placement => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:23.124184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b92ec88-ce6e-4a93-ac3a-7bc626016a76 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:23.231644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b92ec88-ce6e-4a93-ac3a-7bc626016a76 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:23.266145 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:23.266754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:23.267023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:23.267288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:23.267919 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] RBAC: Adding query filter params (name=placement) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:23.268254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:23.269625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:23.275701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 479/976] 10.222.0.39 () {56 vars in 1116 bytes} [Mon Oct 24 08:41:23 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:41:23.281859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:23.384709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c761078-8d51-48d0-8fcb-8b10b4f46cc9 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:23.419541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddb0d1b-0088-43fe-8501-c8a71a4a0d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:23.420169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddb0d1b-0088-43fe-8501-c8a71a4a0d75 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:23.420451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddb0d1b-0088-43fe-8501-c8a71a4a0d75 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:23.420722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddb0d1b-0088-43fe-8501-c8a71a4a0d75 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:23.421420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddb0d1b-0088-43fe-8501-c8a71a4a0d75 None admin] RBAC: Adding query filter params (interface=public, service_id=5443be8bbaad452f82f4571e2c10c077, region_id=RegionOne) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:23.421729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddb0d1b-0088-43fe-8501-c8a71a4a0d75 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:23.423090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddb0d1b-0088-43fe-8501-c8a71a4a0d75 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:23.428802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 498/977] 10.222.0.39 () {56 vars in 1251 bytes} [Mon Oct 24 08:41:23 2022] GET /identity/v3/endpoints?service_id=5443be8bbaad452f82f4571e2c10c077&interface=public®ion_id=RegionOne => generated 194 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:23.435183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:23.439107 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:23.535631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0486b3-efb9-4d5b-b996-fddeb45f12d3 None admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:23.572007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc4008-e732-493e-a650-9a2bb75075b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:23.572765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc4008-e732-493e-a650-9a2bb75075b4 None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:23.572897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc4008-e732-493e-a650-9a2bb75075b4 None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:23.573147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc4008-e732-493e-a650-9a2bb75075b4 None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:23.573741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc4008-e732-493e-a650-9a2bb75075b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:23.575098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc4008-e732-493e-a650-9a2bb75075b4 None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:23.581952 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 480/978] 10.222.0.39 () {56 vars in 1087 bytes} [Mon Oct 24 08:41:23 2022] GET /identity/v3/services => generated 2191 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:41:25.790956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53398420-599e-40cb-ac3a-5126ef2fc351 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:25.791537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53398420-599e-40cb-ac3a-5126ef2fc351 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:25.791537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53398420-599e-40cb-ac3a-5126ef2fc351 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:25.792880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 499/979] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:41:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:41:25.798441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161b50f-00ab-4355-ba46-6250f43473dc None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:25.798876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161b50f-00ab-4355-ba46-6250f43473dc None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:25.798978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161b50f-00ab-4355-ba46-6250f43473dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:25.825454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9161b50f-00ab-4355-ba46-6250f43473dc None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:41:25.827279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9161b50f-00ab-4355-ba46-6250f43473dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:41:25.889835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:25.895863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 481/980] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:41:25 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 24 08:41:26.175744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53398420-599e-40cb-ac3a-5126ef2fc351 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:26.195968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:26.281128 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53398420-599e-40cb-ac3a-5126ef2fc351 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:26.315781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:26.316496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:26.316780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:26.317055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] PATH_INFO: `/v3/services/placement` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:26.317633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:26.318990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 24 08:41:26.328224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 24 08:41:26.329439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.get_or_create( Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application with Lock( Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self._enter() Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application return self.creator() Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application created_value = creator( Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: ERROR keystone.server.flask.application Oct 24 08:41:26.330415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 500/981] 10.222.0.39 () {56 vars in 1117 bytes} [Mon Oct 24 08:41:26 2022] GET /identity/v3/services/placement => generated 90 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 24 08:41:26.337573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9161b50f-00ab-4355-ba46-6250f43473dc None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:26.439621 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9161b50f-00ab-4355-ba46-6250f43473dc None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:26.486679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e143ff75-5c33-4e9a-aed9-2eb3060dd27f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:26.487305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e143ff75-5c33-4e9a-aed9-2eb3060dd27f None admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:26.487582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e143ff75-5c33-4e9a-aed9-2eb3060dd27f None admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:26.487848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e143ff75-5c33-4e9a-aed9-2eb3060dd27f None admin] PATH_INFO: `/v3/services` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:26.488528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e143ff75-5c33-4e9a-aed9-2eb3060dd27f None admin] RBAC: Adding query filter params (name=placement) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:41:26.488827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e143ff75-5c33-4e9a-aed9-2eb3060dd27f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:26.490228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e143ff75-5c33-4e9a-aed9-2eb3060dd27f None admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:26.497546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 482/982] 10.222.0.39 () {56 vars in 1116 bytes} [Mon Oct 24 08:41:26 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:41:26.503502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:41:26.588004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:26.616615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42005dd4-205b-422e-bcf5-55a985a927f8 None admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:41:26.665989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e883dbe-fe32-4bae-9ef1-160ad78e71d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:41:26.666592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e883dbe-fe32-4bae-9ef1-160ad78e71d9 None admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:41:26.666872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e883dbe-fe32-4bae-9ef1-160ad78e71d9 None admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:41:26.667141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e883dbe-fe32-4bae-9ef1-160ad78e71d9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:41:26.667821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e883dbe-fe32-4bae-9ef1-160ad78e71d9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:41:26.669209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e883dbe-fe32-4bae-9ef1-160ad78e71d9 None admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:41:26.688806 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 501/983] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:41:26 2022] POST /identity/v3/endpoints => generated 331 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:41:36.494011 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=69 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:41:36.685532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:35.157444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27f19e2-9d63-47e8-b6f9-567c5e70aaa8 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:35.158098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27f19e2-9d63-47e8-b6f9-567c5e70aaa8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:35.158098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27f19e2-9d63-47e8-b6f9-567c5e70aaa8 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:35.159349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 483/984] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:42:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:35.166217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7f03a-2dfd-4544-a586-8ffce84f8a33 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:35.166785 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7f03a-2dfd-4544-a586-8ffce84f8a33 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:35.166785 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7f03a-2dfd-4544-a586-8ffce84f8a33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:35.274882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-bfa7f03a-2dfd-4544-a586-8ffce84f8a33 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:35.276303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bfa7f03a-2dfd-4544-a586-8ffce84f8a33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:35.511903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 502/985] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:42:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:35.827567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0b1d5d-a6b3-448e-a4ef-2d915d426812 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:35.827845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0b1d5d-a6b3-448e-a4ef-2d915d426812 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:35.828452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0b1d5d-a6b3-448e-a4ef-2d915d426812 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:35.829969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 484/986] 10.222.0.39 () {54 vars in 964 bytes} [Mon Oct 24 08:42:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:35.837103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3954f3d-479c-40d3-94ff-09486cc3d138 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:35.837503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3954f3d-479c-40d3-94ff-09486cc3d138 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:35.837862 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3954f3d-479c-40d3-94ff-09486cc3d138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:35.934435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a3954f3d-479c-40d3-94ff-09486cc3d138 None None] MFA Rules not processed for user `8bfae94e0fa041c9802729370a810245`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:35.935771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a3954f3d-479c-40d3-94ff-09486cc3d138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:35.959448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:36.026772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 503/987] 10.222.0.39 () {58 vars in 1061 bytes} [Mon Oct 24 08:42:35 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:36.035856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0b1d5d-a6b3-448e-a4ef-2d915d426812 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:42:36.039126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0a0b1d5d-a6b3-448e-a4ef-2d915d426812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:36.159073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0b1d5d-a6b3-448e-a4ef-2d915d426812 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:42:36.239703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd16955f-f999-4798-8cf3-51aa70abdd6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:42:36.240780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16955f-f999-4798-8cf3-51aa70abdd6c service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:36.240780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16955f-f999-4798-8cf3-51aa70abdd6c service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:36.240780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16955f-f999-4798-8cf3-51aa70abdd6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:36.242410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-fd16955f-f999-4798-8cf3-51aa70abdd6c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:36.302761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:36.369248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd16955f-f999-4798-8cf3-51aa70abdd6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:42:36.370742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd16955f-f999-4798-8cf3-51aa70abdd6c service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:42:36.395443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 485/988] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:42:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:42:38.897666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2483038-8c6a-4b3a-b732-d389128e3585 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:38.898215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2483038-8c6a-4b3a-b732-d389128e3585 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:38.898310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2483038-8c6a-4b3a-b732-d389128e3585 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:38.899709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 504/989] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:42:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:38.905498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036a892-658e-4320-849d-17a668a43855 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:38.905972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036a892-658e-4320-849d-17a668a43855 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:38.906067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036a892-658e-4320-849d-17a668a43855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:38.982597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a036a892-658e-4320-849d-17a668a43855 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:38.983813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a036a892-658e-4320-849d-17a668a43855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:39.087962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 486/990] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:42:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:39.379089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2483038-8c6a-4b3a-b732-d389128e3585 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:42:39.381329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e2483038-8c6a-4b3a-b732-d389128e3585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:39.503314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2483038-8c6a-4b3a-b732-d389128e3585 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:42:39.587667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:39.590431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0028fdd4-7bbc-4de4-8d07-5bcfad487b71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:42:39.591228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0028fdd4-7bbc-4de4-8d07-5bcfad487b71 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:39.591542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0028fdd4-7bbc-4de4-8d07-5bcfad487b71 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:39.592017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0028fdd4-7bbc-4de4-8d07-5bcfad487b71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:39.593969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0028fdd4-7bbc-4de4-8d07-5bcfad487b71 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:39.707735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0028fdd4-7bbc-4de4-8d07-5bcfad487b71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:42:39.709866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0028fdd4-7bbc-4de4-8d07-5bcfad487b71 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:42:39.729360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 505/991] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:42:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:42:42.287273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f90610-ab7f-47ec-90df-3bcdffb1e76d None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:42.288033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f90610-ab7f-47ec-90df-3bcdffb1e76d None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:42.288033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f90610-ab7f-47ec-90df-3bcdffb1e76d None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:42.289117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 487/992] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:42:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:42.295157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9de03b-a5b0-4f37-9a7a-dd7236270929 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:42.295692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9de03b-a5b0-4f37-9a7a-dd7236270929 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:42.295782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9de03b-a5b0-4f37-9a7a-dd7236270929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:42.362426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6e9de03b-a5b0-4f37-9a7a-dd7236270929 None None] MFA Rules not processed for user `97b89b4a23fb40a2bb9cfec9d4653ce7`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:42.363620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6e9de03b-a5b0-4f37-9a7a-dd7236270929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:42.415716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:42.461821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 506/993] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:42:42 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:42.750987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f90610-ab7f-47ec-90df-3bcdffb1e76d None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:42:42.752533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d5f90610-ab7f-47ec-90df-3bcdffb1e76d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:42.762714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:42.857189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f90610-ab7f-47ec-90df-3bcdffb1e76d None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:42:42.931585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344d367a-7101-4fc5-9dd0-8a83ef00959c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:42:42.932199 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344d367a-7101-4fc5-9dd0-8a83ef00959c service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:42.932479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344d367a-7101-4fc5-9dd0-8a83ef00959c service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:42.932748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344d367a-7101-4fc5-9dd0-8a83ef00959c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:42.934326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-344d367a-7101-4fc5-9dd0-8a83ef00959c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:43.000272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:43.048732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344d367a-7101-4fc5-9dd0-8a83ef00959c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:42:43.050167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344d367a-7101-4fc5-9dd0-8a83ef00959c service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:42:43.068155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 488/994] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:42:42 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:42:45.832782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ea8fa-2fd3-411b-9551-1e2fef384645 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:45.833323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ea8fa-2fd3-411b-9551-1e2fef384645 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:45.833323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ea8fa-2fd3-411b-9551-1e2fef384645 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:45.835535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 507/995] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:42:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:45.841265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c5fc4c-a427-4845-b165-6904112d4b9a None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:45.841665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c5fc4c-a427-4845-b165-6904112d4b9a None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:45.841829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c5fc4c-a427-4845-b165-6904112d4b9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:45.908773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-92c5fc4c-a427-4845-b165-6904112d4b9a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:45.909996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-92c5fc4c-a427-4845-b165-6904112d4b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:46.017018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 489/996] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:42:45 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:46.310196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5ea8fa-2fd3-411b-9551-1e2fef384645 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:42:46.311715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-cb5ea8fa-2fd3-411b-9551-1e2fef384645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:46.414327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5ea8fa-2fd3-411b-9551-1e2fef384645 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:42:46.489851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8547063-b521-4af5-9c1d-67f9b9569385 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:42:46.490611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8547063-b521-4af5-9c1d-67f9b9569385 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:46.490896 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8547063-b521-4af5-9c1d-67f9b9569385 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:46.491167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8547063-b521-4af5-9c1d-67f9b9569385 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:46.492804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d8547063-b521-4af5-9c1d-67f9b9569385 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:46.500146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:46.615967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8547063-b521-4af5-9c1d-67f9b9569385 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:42:46.617333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8547063-b521-4af5-9c1d-67f9b9569385 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:42:46.637287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 508/997] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:42:46 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:42:49.555916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea82c8-1e41-4ac8-b502-80adc3091aac None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:49.556481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea82c8-1e41-4ac8-b502-80adc3091aac None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:49.556481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea82c8-1e41-4ac8-b502-80adc3091aac None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:49.559733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 490/998] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:42:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:49.564108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006e8a1f-4fb4-404f-a73c-44572f3d5563 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:49.564690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006e8a1f-4fb4-404f-a73c-44572f3d5563 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:49.564690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006e8a1f-4fb4-404f-a73c-44572f3d5563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:49.643158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-006e8a1f-4fb4-404f-a73c-44572f3d5563 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:49.644434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-006e8a1f-4fb4-404f-a73c-44572f3d5563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:49.713111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:49.758776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 509/999] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:42:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:50.047201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbea82c8-1e41-4ac8-b502-80adc3091aac None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:42:50.048836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cbea82c8-1e41-4ac8-b502-80adc3091aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:50.069344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:50.154653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbea82c8-1e41-4ac8-b502-80adc3091aac None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:42:50.230338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4dc17-3367-4b8a-a3a5-fc712a5151c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:42:50.230984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4dc17-3367-4b8a-a3a5-fc712a5151c3 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:50.231268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4dc17-3367-4b8a-a3a5-fc712a5151c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:50.231544 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4dc17-3367-4b8a-a3a5-fc712a5151c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:50.233242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9aa4dc17-3367-4b8a-a3a5-fc712a5151c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:50.315184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:50.355394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa4dc17-3367-4b8a-a3a5-fc712a5151c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:42:50.356864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa4dc17-3367-4b8a-a3a5-fc712a5151c3 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:42:50.377588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 491/1000] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:42:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:42:52.698416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27764d-abf2-4dfa-839a-e8faf1063f33 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:52.698950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27764d-abf2-4dfa-839a-e8faf1063f33 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:52.698950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27764d-abf2-4dfa-839a-e8faf1063f33 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:52.700368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 510/1001] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:42:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:52.705939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bb9217-b1fa-42c2-90bb-da97f40bd0e0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:52.706407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bb9217-b1fa-42c2-90bb-da97f40bd0e0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:52.706498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bb9217-b1fa-42c2-90bb-da97f40bd0e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:52.791793 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c5bb9217-b1fa-42c2-90bb-da97f40bd0e0 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:52.793097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c5bb9217-b1fa-42c2-90bb-da97f40bd0e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:52.900240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 492/1002] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:42:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:53.190897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e27764d-abf2-4dfa-839a-e8faf1063f33 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:42:53.192425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7e27764d-abf2-4dfa-839a-e8faf1063f33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:53.299204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e27764d-abf2-4dfa-839a-e8faf1063f33 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:42:53.378409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f236f6b2-6f9c-49c1-b079-87ad987be7de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:42:53.379038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f236f6b2-6f9c-49c1-b079-87ad987be7de service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:53.379279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f236f6b2-6f9c-49c1-b079-87ad987be7de service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:53.379646 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f236f6b2-6f9c-49c1-b079-87ad987be7de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:53.382258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f236f6b2-6f9c-49c1-b079-87ad987be7de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:53.401172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:53.500478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f236f6b2-6f9c-49c1-b079-87ad987be7de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:42:53.501812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f236f6b2-6f9c-49c1-b079-87ad987be7de service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:42:53.520837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 511/1003] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:42:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:42:55.787933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3641d57-501b-4a9d-bc5a-a9e63c6f16a6 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:55.787933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3641d57-501b-4a9d-bc5a-a9e63c6f16a6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:55.787933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3641d57-501b-4a9d-bc5a-a9e63c6f16a6 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:55.789200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 493/1004] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:42:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:55.794717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f200a307-4c2a-419f-b2fb-930d00861394 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:55.795201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f200a307-4c2a-419f-b2fb-930d00861394 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:55.795305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f200a307-4c2a-419f-b2fb-930d00861394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:55.866221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f200a307-4c2a-419f-b2fb-930d00861394 None None] MFA Rules not processed for user `97b89b4a23fb40a2bb9cfec9d4653ce7`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:55.867435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f200a307-4c2a-419f-b2fb-930d00861394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:55.934115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:55.967667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 512/1005] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:42:55 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:56.252640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3641d57-501b-4a9d-bc5a-a9e63c6f16a6 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:42:56.254197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f3641d57-501b-4a9d-bc5a-a9e63c6f16a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:56.289921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:56.368988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3641d57-501b-4a9d-bc5a-a9e63c6f16a6 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:42:56.448446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd1c7ef-19c4-4e48-8705-6cd2a412a0bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:42:56.449046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd1c7ef-19c4-4e48-8705-6cd2a412a0bc service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:56.449326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd1c7ef-19c4-4e48-8705-6cd2a412a0bc service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:56.449593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd1c7ef-19c4-4e48-8705-6cd2a412a0bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:56.451182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0bd1c7ef-19c4-4e48-8705-6cd2a412a0bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:56.537027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:56.574110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd1c7ef-19c4-4e48-8705-6cd2a412a0bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:42:56.575547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd1c7ef-19c4-4e48-8705-6cd2a412a0bc service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:42:56.597769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 494/1006] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:42:56 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:42:59.123271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ac1bb-8004-4da4-91f1-543d00a22c97 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:59.123784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ac1bb-8004-4da4-91f1-543d00a22c97 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:59.123784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ac1bb-8004-4da4-91f1-543d00a22c97 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:59.125171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 513/1007] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:42:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:59.131010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4269bf1-f458-4058-b9a7-b91c9abb7e68 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:59.131412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4269bf1-f458-4058-b9a7-b91c9abb7e68 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:59.131562 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4269bf1-f458-4058-b9a7-b91c9abb7e68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:59.221097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c4269bf1-f458-4058-b9a7-b91c9abb7e68 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:59.223175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c4269bf1-f458-4058-b9a7-b91c9abb7e68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:59.325760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 495/1008] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:42:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:42:59.647157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8701e4c-6903-4cc0-bcb7-8479954b272e None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:59.647442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8701e4c-6903-4cc0-bcb7-8479954b272e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:59.647719 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8701e4c-6903-4cc0-bcb7-8479954b272e None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:59.649107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 514/1009] 10.222.0.39 () {54 vars in 964 bytes} [Mon Oct 24 08:42:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:42:59.655093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beba8988-67fb-4d58-8f08-3c00fcefc3f6 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:42:59.655389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beba8988-67fb-4d58-8f08-3c00fcefc3f6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:42:59.655668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beba8988-67fb-4d58-8f08-3c00fcefc3f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:42:59.701908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:42:59.734436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-beba8988-67fb-4d58-8f08-3c00fcefc3f6 None None] MFA Rules not processed for user `8bfae94e0fa041c9802729370a810245`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:42:59.735662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-beba8988-67fb-4d58-8f08-3c00fcefc3f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:59.812776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 496/1010] 10.222.0.39 () {58 vars in 1061 bytes} [Mon Oct 24 08:42:59 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:42:59.822210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8701e4c-6903-4cc0-bcb7-8479954b272e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:42:59.823717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b8701e4c-6903-4cc0-bcb7-8479954b272e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:42:59.931547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8701e4c-6903-4cc0-bcb7-8479954b272e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:00.015065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3026a3c5-95f8-4f7c-9fce-614336261ef4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:00.015672 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3026a3c5-95f8-4f7c-9fce-614336261ef4 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:00.015956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3026a3c5-95f8-4f7c-9fce-614336261ef4 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:00.016272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3026a3c5-95f8-4f7c-9fce-614336261ef4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:00.017850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3026a3c5-95f8-4f7c-9fce-614336261ef4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:00.048720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:00.134344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3026a3c5-95f8-4f7c-9fce-614336261ef4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:00.135767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3026a3c5-95f8-4f7c-9fce-614336261ef4 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:00.154550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 515/1011] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:42:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:02.640067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eca1d3-97dd-4578-ace0-f8776b050c2e None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:02.640632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eca1d3-97dd-4578-ace0-f8776b050c2e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:02.640632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eca1d3-97dd-4578-ace0-f8776b050c2e None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:02.642001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 497/1012] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:02.648409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bd56ad-7366-4b36-a4a1-cc2202a32093 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:02.648866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bd56ad-7366-4b36-a4a1-cc2202a32093 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:02.648959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bd56ad-7366-4b36-a4a1-cc2202a32093 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:02.715057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-02bd56ad-7366-4b36-a4a1-cc2202a32093 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:02.716292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-02bd56ad-7366-4b36-a4a1-cc2202a32093 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:02.794178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:02.816085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 516/1013] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:03.103161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eca1d3-97dd-4578-ace0-f8776b050c2e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:03.104729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-06eca1d3-97dd-4578-ace0-f8776b050c2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:03.211288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eca1d3-97dd-4578-ace0-f8776b050c2e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:03.240991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:03.289119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954fed83-2111-441a-8fa1-ab22061fa76b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:03.289682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fed83-2111-441a-8fa1-ab22061fa76b service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:03.289932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fed83-2111-441a-8fa1-ab22061fa76b service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:03.290226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fed83-2111-441a-8fa1-ab22061fa76b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:03.291955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-954fed83-2111-441a-8fa1-ab22061fa76b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:03.405804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954fed83-2111-441a-8fa1-ab22061fa76b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:03.407390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954fed83-2111-441a-8fa1-ab22061fa76b service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:03.426230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 498/1014] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:05.958261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa63fa50-8610-4d16-bac9-624b7170b7ca None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:05.958865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa63fa50-8610-4d16-bac9-624b7170b7ca None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:05.958865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa63fa50-8610-4d16-bac9-624b7170b7ca None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:05.960087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 517/1015] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:05.965506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640aec7-eb00-48a8-a92c-6b285e1c5952 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:05.965973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640aec7-eb00-48a8-a92c-6b285e1c5952 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:05.966068 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640aec7-eb00-48a8-a92c-6b285e1c5952 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:06.039222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-1640aec7-eb00-48a8-a92c-6b285e1c5952 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:06.040461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1640aec7-eb00-48a8-a92c-6b285e1c5952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:06.048969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:06.140865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 499/1016] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:05 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:06.431885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa63fa50-8610-4d16-bac9-624b7170b7ca None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:06.433447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-aa63fa50-8610-4d16-bac9-624b7170b7ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:06.540105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa63fa50-8610-4d16-bac9-624b7170b7ca None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:06.616954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf02d059-b114-4039-b1b3-85253f77f14f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:06.617557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf02d059-b114-4039-b1b3-85253f77f14f service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:06.617839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf02d059-b114-4039-b1b3-85253f77f14f service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:06.618113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf02d059-b114-4039-b1b3-85253f77f14f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:06.619694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-cf02d059-b114-4039-b1b3-85253f77f14f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:06.655523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:06.731636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf02d059-b114-4039-b1b3-85253f77f14f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:06.733007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf02d059-b114-4039-b1b3-85253f77f14f service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:06.751759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 518/1017] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:06 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:11.314607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96798113-055b-41d3-b88f-d424813fe4c7 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:11.315114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96798113-055b-41d3-b88f-d424813fe4c7 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:11.315114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96798113-055b-41d3-b88f-d424813fe4c7 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:11.316507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 500/1018] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:11.322910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be554560-e80f-44bd-9f1f-e9bc03857e48 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:11.323412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be554560-e80f-44bd-9f1f-e9bc03857e48 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:11.323504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be554560-e80f-44bd-9f1f-e9bc03857e48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:11.395771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-be554560-e80f-44bd-9f1f-e9bc03857e48 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:11.397015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-be554560-e80f-44bd-9f1f-e9bc03857e48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:11.489795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:11.502252 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 519/1019] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:11.791980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96798113-055b-41d3-b88f-d424813fe4c7 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:11.793584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-96798113-055b-41d3-b88f-d424813fe4c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:11.906353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96798113-055b-41d3-b88f-d424813fe4c7 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:11.949863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:12.000763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00da90b-4371-4906-9b9f-24f9996c235e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:12.001378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00da90b-4371-4906-9b9f-24f9996c235e service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:12.001655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00da90b-4371-4906-9b9f-24f9996c235e service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:12.001923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00da90b-4371-4906-9b9f-24f9996c235e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:12.003519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f00da90b-4371-4906-9b9f-24f9996c235e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:12.121535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00da90b-4371-4906-9b9f-24f9996c235e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:12.123223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00da90b-4371-4906-9b9f-24f9996c235e service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:12.144242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 501/1020] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:43:14.996790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f8c631-1987-4663-bed1-b8e022ee8a36 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:14.997264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f8c631-1987-4663-bed1-b8e022ee8a36 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:14.997357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f8c631-1987-4663-bed1-b8e022ee8a36 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:14.998939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 520/1021] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:15.004331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d42dbff-6053-4b75-90c1-f76987468a6d None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:15.005270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d42dbff-6053-4b75-90c1-f76987468a6d None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:15.005440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d42dbff-6053-4b75-90c1-f76987468a6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:15.074823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-6d42dbff-6053-4b75-90c1-f76987468a6d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:15.076033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6d42dbff-6053-4b75-90c1-f76987468a6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:15.094365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:15.179660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 502/1022] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:15 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:15.469816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f8c631-1987-4663-bed1-b8e022ee8a36 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:15.471372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-62f8c631-1987-4663-bed1-b8e022ee8a36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:15.593161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f8c631-1987-4663-bed1-b8e022ee8a36 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:15.676650 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafbf767-961e-4358-828c-90f6494a34df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:15.677080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafbf767-961e-4358-828c-90f6494a34df service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:15.677357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafbf767-961e-4358-828c-90f6494a34df service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:15.677624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafbf767-961e-4358-828c-90f6494a34df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:15.679237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-eafbf767-961e-4358-828c-90f6494a34df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:15.724723 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:15.795958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafbf767-961e-4358-828c-90f6494a34df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:15.797332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafbf767-961e-4358-828c-90f6494a34df service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:15.817018 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 521/1023] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:15 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:20.403146 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfd19ab-96f1-4752-9221-3621f32a60ca None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:20.403631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfd19ab-96f1-4752-9221-3621f32a60ca None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:20.403736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfd19ab-96f1-4752-9221-3621f32a60ca None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:20.405106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 503/1024] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:20.410758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27874b76-f9c0-45e6-ae8d-8272a44e97e8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:20.411194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27874b76-f9c0-45e6-ae8d-8272a44e97e8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:20.411378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27874b76-f9c0-45e6-ae8d-8272a44e97e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:20.485276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-27874b76-f9c0-45e6-ae8d-8272a44e97e8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:20.486496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-27874b76-f9c0-45e6-ae8d-8272a44e97e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:20.590910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:20.594951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 522/1025] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:20.885696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfd19ab-96f1-4752-9221-3621f32a60ca None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:20.887317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0cfd19ab-96f1-4752-9221-3621f32a60ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:21.008390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfd19ab-96f1-4752-9221-3621f32a60ca None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:21.063689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:21.094579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f68a06-c8f6-4ff7-a8d4-e4d27b92ef5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:21.095178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f68a06-c8f6-4ff7-a8d4-e4d27b92ef5b service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:21.095451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f68a06-c8f6-4ff7-a8d4-e4d27b92ef5b service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:21.095717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f68a06-c8f6-4ff7-a8d4-e4d27b92ef5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:21.097340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-53f68a06-c8f6-4ff7-a8d4-e4d27b92ef5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:21.212676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f68a06-c8f6-4ff7-a8d4-e4d27b92ef5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:21.214138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f68a06-c8f6-4ff7-a8d4-e4d27b92ef5b service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:21.234148 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 504/1026] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:26.330913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2b8d3-4659-49d5-8015-99926f1a9113 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:26.331376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2b8d3-4659-49d5-8015-99926f1a9113 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:26.331376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2b8d3-4659-49d5-8015-99926f1a9113 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:26.333547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 523/1027] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:43:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:26.339650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3600ac70-0ca7-4e56-88cf-6ad4d8473aaf None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:26.340448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3600ac70-0ca7-4e56-88cf-6ad4d8473aaf None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:26.340945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3600ac70-0ca7-4e56-88cf-6ad4d8473aaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:26.502515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3600ac70-0ca7-4e56-88cf-6ad4d8473aaf None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:26.503802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3600ac70-0ca7-4e56-88cf-6ad4d8473aaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:26.529663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:26.627354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 505/1028] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:43:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:31.120238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a6c23-42b2-4672-a42a-b2876810d783 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:31.120784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a6c23-42b2-4672-a42a-b2876810d783 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:31.120784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a6c23-42b2-4672-a42a-b2876810d783 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:31.122122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 524/1029] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:43:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:31.127804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd145013-17ac-44ac-aa1d-676d800c8136 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:31.128307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd145013-17ac-44ac-aa1d-676d800c8136 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:31.128529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd145013-17ac-44ac-aa1d-676d800c8136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:31.210791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-bd145013-17ac-44ac-aa1d-676d800c8136 None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:31.212111 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bd145013-17ac-44ac-aa1d-676d800c8136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:31.309214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:31.359999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 506/1030] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:43:31 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:32.690111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19fba5-7276-4ad9-95c2-5af189639456 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:32.690885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19fba5-7276-4ad9-95c2-5af189639456 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:32.690885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19fba5-7276-4ad9-95c2-5af189639456 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:32.693017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 525/1031] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:43:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:32.700894 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d2afb1-9389-4004-af5f-98b34a3d6f15 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:32.701477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d2afb1-9389-4004-af5f-98b34a3d6f15 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:32.701858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d2afb1-9389-4004-af5f-98b34a3d6f15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:32.810085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-23d2afb1-9389-4004-af5f-98b34a3d6f15 None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:32.811302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-23d2afb1-9389-4004-af5f-98b34a3d6f15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:32.971180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 507/1032] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:43:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:36.537480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19f763a-b134-4a56-9947-fc5c58e8f617 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:36.538073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19f763a-b134-4a56-9947-fc5c58e8f617 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:36.538073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19f763a-b134-4a56-9947-fc5c58e8f617 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:36.539399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 526/1033] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:36.545186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b5bba-f54d-4443-bc3c-6091f48564f3 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:36.545672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b5bba-f54d-4443-bc3c-6091f48564f3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:36.545783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b5bba-f54d-4443-bc3c-6091f48564f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:36.609040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:36.641493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-bc4b5bba-f54d-4443-bc3c-6091f48564f3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:36.642976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bc4b5bba-f54d-4443-bc3c-6091f48564f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:36.773742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 508/1034] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:37.076476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54bd0a4-02da-47b4-9148-4e4b0a56bbd7 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:37.077182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54bd0a4-02da-47b4-9148-4e4b0a56bbd7 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:37.077675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54bd0a4-02da-47b4-9148-4e4b0a56bbd7 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:37.079578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 527/1035] 10.222.0.39 () {54 vars in 961 bytes} [Mon Oct 24 08:43:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:37.092893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c700ffb2-9831-4eca-8a3f-7e80174549a8 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:37.093256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c700ffb2-9831-4eca-8a3f-7e80174549a8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:37.093439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c700ffb2-9831-4eca-8a3f-7e80174549a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:37.203360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c700ffb2-9831-4eca-8a3f-7e80174549a8 None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:37.204668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c700ffb2-9831-4eca-8a3f-7e80174549a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:37.257967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:37.330948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 509/1036] 10.222.0.39 () {58 vars in 1058 bytes} [Mon Oct 24 08:43:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:37.339056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54bd0a4-02da-47b4-9148-4e4b0a56bbd7 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:37.340628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a54bd0a4-02da-47b4-9148-4e4b0a56bbd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:37.506316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54bd0a4-02da-47b4-9148-4e4b0a56bbd7 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:37.602723 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:37.673254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21fc6ae-9e0c-4d49-829d-7f0f86b5646e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:37.674254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21fc6ae-9e0c-4d49-829d-7f0f86b5646e service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:37.674740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21fc6ae-9e0c-4d49-829d-7f0f86b5646e service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:37.675222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21fc6ae-9e0c-4d49-829d-7f0f86b5646e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:37.678798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d21fc6ae-9e0c-4d49-829d-7f0f86b5646e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:37.789299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21fc6ae-9e0c-4d49-829d-7f0f86b5646e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:37.790962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21fc6ae-9e0c-4d49-829d-7f0f86b5646e service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:37.813989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 528/1037] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:43:40.313625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b487ad-c906-448b-a357-ca2e68c412d6 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:40.313625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b487ad-c906-448b-a357-ca2e68c412d6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:40.314366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b487ad-c906-448b-a357-ca2e68c412d6 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:40.323720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60e6c86-4798-430f-a6ce-db5a5cfa907a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:40.323720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60e6c86-4798-430f-a6ce-db5a5cfa907a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:40.323720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60e6c86-4798-430f-a6ce-db5a5cfa907a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:40.326202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 510/1038] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:40 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:40.404608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b60e6c86-4798-430f-a6ce-db5a5cfa907a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:40.406173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b60e6c86-4798-430f-a6ce-db5a5cfa907a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:40.448454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:40.515779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 529/1039] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:40.808108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b487ad-c906-448b-a357-ca2e68c412d6 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:40.809641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d2b487ad-c906-448b-a357-ca2e68c412d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:41.000908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b487ad-c906-448b-a357-ca2e68c412d6 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:41.013457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:41.139378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b37572-7c60-49ca-a620-edd8c79c1011 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:41.140425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b37572-7c60-49ca-a620-edd8c79c1011 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:41.140710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b37572-7c60-49ca-a620-edd8c79c1011 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:41.140993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b37572-7c60-49ca-a620-edd8c79c1011 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:41.142610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d7b37572-7c60-49ca-a620-edd8c79c1011 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:41.239465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b37572-7c60-49ca-a620-edd8c79c1011 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:41.241188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b37572-7c60-49ca-a620-edd8c79c1011 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:41.260972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 511/1040] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:43.812298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f656c297-d62a-4285-b513-77b2f6cd968c None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:43.813291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f656c297-d62a-4285-b513-77b2f6cd968c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:43.813792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f656c297-d62a-4285-b513-77b2f6cd968c None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:43.815413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 530/1041] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:43.821711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de0bb1-bd86-435d-85b9-cda6cc90ad99 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:43.822256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de0bb1-bd86-435d-85b9-cda6cc90ad99 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:43.822456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de0bb1-bd86-435d-85b9-cda6cc90ad99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:43.839560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:43.900855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f0de0bb1-bd86-435d-85b9-cda6cc90ad99 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:43.902365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f0de0bb1-bd86-435d-85b9-cda6cc90ad99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:44.018255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 512/1042] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:44.320404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf02508-67eb-4ddc-bdae-d01ea9869012 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:44.320816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf02508-67eb-4ddc-bdae-d01ea9869012 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:44.321178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf02508-67eb-4ddc-bdae-d01ea9869012 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:44.322801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 531/1043] 10.222.0.39 () {54 vars in 961 bytes} [Mon Oct 24 08:43:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:44.328316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76c563-220f-46c6-a998-82e63e228180 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:44.328594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76c563-220f-46c6-a998-82e63e228180 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:44.328877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76c563-220f-46c6-a998-82e63e228180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:44.410676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-0d76c563-220f-46c6-a998-82e63e228180 None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:44.411909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0d76c563-220f-46c6-a998-82e63e228180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:44.449607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:44.544367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 513/1044] 10.222.0.39 () {58 vars in 1058 bytes} [Mon Oct 24 08:43:44 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:44.552926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf02508-67eb-4ddc-bdae-d01ea9869012 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:44.554639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ebf02508-67eb-4ddc-bdae-d01ea9869012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:44.720804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf02508-67eb-4ddc-bdae-d01ea9869012 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:44.742893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:44.845947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febd21f6-e23b-4d99-a9b1-ffb975a5f85d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:44.846582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd21f6-e23b-4d99-a9b1-ffb975a5f85d service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:44.846856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd21f6-e23b-4d99-a9b1-ffb975a5f85d service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:44.847083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd21f6-e23b-4d99-a9b1-ffb975a5f85d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:44.849216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-febd21f6-e23b-4d99-a9b1-ffb975a5f85d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:44.962695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febd21f6-e23b-4d99-a9b1-ffb975a5f85d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:44.964380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febd21f6-e23b-4d99-a9b1-ffb975a5f85d service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:44.995782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 532/1045] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:45.066720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761e3c92-ffff-4b3b-b148-297c261b467e None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:45.067197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761e3c92-ffff-4b3b-b148-297c261b467e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:45.067469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761e3c92-ffff-4b3b-b148-297c261b467e None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:45.068845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 514/1046] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:45.074229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b2cc5b-7db5-4417-89dc-e4a4d1a32ae6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:45.074522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b2cc5b-7db5-4417-89dc-e4a4d1a32ae6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:45.074801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b2cc5b-7db5-4417-89dc-e4a4d1a32ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:45.110132 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:45.165098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c0b2cc5b-7db5-4417-89dc-e4a4d1a32ae6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:45.166463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c0b2cc5b-7db5-4417-89dc-e4a4d1a32ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:45.296264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 533/1047] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:45.637164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057bd83-6681-4147-8b1a-992f3c4b58a3 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:45.637164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057bd83-6681-4147-8b1a-992f3c4b58a3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:45.637164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057bd83-6681-4147-8b1a-992f3c4b58a3 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:45.638535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 515/1048] 10.222.0.39 () {54 vars in 963 bytes} [Mon Oct 24 08:43:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:45.643475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5847b713-cec2-4216-bee4-3a6aef434d98 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:45.643759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5847b713-cec2-4216-bee4-3a6aef434d98 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:45.644024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5847b713-cec2-4216-bee4-3a6aef434d98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:45.734126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5847b713-cec2-4216-bee4-3a6aef434d98 None None] MFA Rules not processed for user `412f520f90564e44819e8bf947992efb`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:45.735317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5847b713-cec2-4216-bee4-3a6aef434d98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:45.787872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:45.826956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 534/1049] 10.222.0.39 () {58 vars in 1060 bytes} [Mon Oct 24 08:43:45 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:45.836016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3057bd83-6681-4147-8b1a-992f3c4b58a3 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:45.837760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3057bd83-6681-4147-8b1a-992f3c4b58a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:45.963075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3057bd83-6681-4147-8b1a-992f3c4b58a3 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:46.036402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:46.086290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf90a610-641a-4fb6-b6d6-e2aa7a4a4999 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:46.086290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90a610-641a-4fb6-b6d6-e2aa7a4a4999 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:46.086290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90a610-641a-4fb6-b6d6-e2aa7a4a4999 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:46.086290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90a610-641a-4fb6-b6d6-e2aa7a4a4999 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:46.087518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bf90a610-641a-4fb6-b6d6-e2aa7a4a4999 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:46.239144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf90a610-641a-4fb6-b6d6-e2aa7a4a4999 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:46.241160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf90a610-641a-4fb6-b6d6-e2aa7a4a4999 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:46.269765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 516/1050] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:43:47.308950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef62203-edb7-4ed6-8341-e1f374011824 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:47.309524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef62203-edb7-4ed6-8341-e1f374011824 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:47.309524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef62203-edb7-4ed6-8341-e1f374011824 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:47.310835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 535/1051] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:47.316035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f305c59e-7a09-4bf8-9efc-30af951ae895 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:47.316953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f305c59e-7a09-4bf8-9efc-30af951ae895 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:47.317270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f305c59e-7a09-4bf8-9efc-30af951ae895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:47.385790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f305c59e-7a09-4bf8-9efc-30af951ae895 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:47.386995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f305c59e-7a09-4bf8-9efc-30af951ae895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:47.408631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:47.503632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 517/1052] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:47.799818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef62203-edb7-4ed6-8341-e1f374011824 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:47.801559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7ef62203-edb7-4ed6-8341-e1f374011824 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:47.974205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef62203-edb7-4ed6-8341-e1f374011824 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:48.037603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:48.103257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d927346-3eb2-41fc-a2b2-1adeda36887b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:48.103932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d927346-3eb2-41fc-a2b2-1adeda36887b service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:48.104304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d927346-3eb2-41fc-a2b2-1adeda36887b service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:48.104638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d927346-3eb2-41fc-a2b2-1adeda36887b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:48.106360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6d927346-3eb2-41fc-a2b2-1adeda36887b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:48.207540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d927346-3eb2-41fc-a2b2-1adeda36887b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:48.209201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d927346-3eb2-41fc-a2b2-1adeda36887b service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:48.228361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 536/1053] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:50.525179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0699e839-ed3e-47ce-8488-28245c5a18e3 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:50.525717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0699e839-ed3e-47ce-8488-28245c5a18e3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:50.525717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0699e839-ed3e-47ce-8488-28245c5a18e3 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:50.527217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 518/1054] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:50.532495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e2cc80-e82a-45ac-bfaf-9972394b3e63 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:50.532926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e2cc80-e82a-45ac-bfaf-9972394b3e63 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:50.533030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e2cc80-e82a-45ac-bfaf-9972394b3e63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:50.604791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b7e2cc80-e82a-45ac-bfaf-9972394b3e63 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:50.605990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b7e2cc80-e82a-45ac-bfaf-9972394b3e63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:50.613513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:50.719368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 537/1055] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:51.012552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0699e839-ed3e-47ce-8488-28245c5a18e3 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:51.014756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0699e839-ed3e-47ce-8488-28245c5a18e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:51.176985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0699e839-ed3e-47ce-8488-28245c5a18e3 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:51.180770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:51.299037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eb86f2-dc5a-4c0c-a9d8-bdae845e951d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:51.299748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb86f2-dc5a-4c0c-a9d8-bdae845e951d service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:51.300027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb86f2-dc5a-4c0c-a9d8-bdae845e951d service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:51.300333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb86f2-dc5a-4c0c-a9d8-bdae845e951d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:51.302672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d1eb86f2-dc5a-4c0c-a9d8-bdae845e951d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:51.398998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1eb86f2-dc5a-4c0c-a9d8-bdae845e951d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:51.400642 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1eb86f2-dc5a-4c0c-a9d8-bdae845e951d service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:51.420890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 519/1056] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:43:53.684686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d7da7a-70f5-40a6-ae23-d31da6848931 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:53.685276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d7da7a-70f5-40a6-ae23-d31da6848931 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:53.685276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d7da7a-70f5-40a6-ae23-d31da6848931 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:53.688179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 538/1057] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:53.692740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b40bd-e808-475a-b733-82aea8b2b1c1 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:53.693224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b40bd-e808-475a-b733-82aea8b2b1c1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:53.693325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b40bd-e808-475a-b733-82aea8b2b1c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:53.705661 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:53.771351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-897b40bd-e808-475a-b733-82aea8b2b1c1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:53.772715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-897b40bd-e808-475a-b733-82aea8b2b1c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:53.888388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 520/1058] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:54.202772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d7da7a-70f5-40a6-ae23-d31da6848931 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:54.204776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-01d7da7a-70f5-40a6-ae23-d31da6848931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:54.362290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:54.374858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d7da7a-70f5-40a6-ae23-d31da6848931 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:54.504406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32230850-ae90-4e1a-b706-94a7edc5a979 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:54.505478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32230850-ae90-4e1a-b706-94a7edc5a979 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:54.505752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32230850-ae90-4e1a-b706-94a7edc5a979 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:54.506023 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32230850-ae90-4e1a-b706-94a7edc5a979 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:54.507604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-32230850-ae90-4e1a-b706-94a7edc5a979 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:54.608146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32230850-ae90-4e1a-b706-94a7edc5a979 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:54.609759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32230850-ae90-4e1a-b706-94a7edc5a979 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:54.621360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:54.631078 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 539/1059] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:43:56.929834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58592de9-a027-48bc-9394-8623b82d4a3e None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:56.930378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58592de9-a027-48bc-9394-8623b82d4a3e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:56.930641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58592de9-a027-48bc-9394-8623b82d4a3e None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:56.932504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 521/1060] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:43:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:43:56.938072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00728660-7063-487e-8e01-2a3f9a6f8f5d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:56.938804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00728660-7063-487e-8e01-2a3f9a6f8f5d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:56.938804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00728660-7063-487e-8e01-2a3f9a6f8f5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:57.016408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-00728660-7063-487e-8e01-2a3f9a6f8f5d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:43:57.017663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-00728660-7063-487e-8e01-2a3f9a6f8f5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:57.125575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 540/1061] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:43:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:43:57.413535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58592de9-a027-48bc-9394-8623b82d4a3e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:43:57.415504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-58592de9-a027-48bc-9394-8623b82d4a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:57.527662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:57.595475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58592de9-a027-48bc-9394-8623b82d4a3e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:43:57.720822 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d04792-7337-43ef-b913-f9b5f4511755 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:43:57.721577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d04792-7337-43ef-b913-f9b5f4511755 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:43:57.722014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d04792-7337-43ef-b913-f9b5f4511755 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:43:57.723084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d04792-7337-43ef-b913-f9b5f4511755 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:43:57.725123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d9d04792-7337-43ef-b913-f9b5f4511755 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:43:57.790875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:43:57.829942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d04792-7337-43ef-b913-f9b5f4511755 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:43:57.831537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d04792-7337-43ef-b913-f9b5f4511755 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:43:57.854714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 522/1062] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:43:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:00.177230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fe521-dfc3-4afe-ba0c-1fd2022f7f35 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:00.178837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fe521-dfc3-4afe-ba0c-1fd2022f7f35 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:00.178837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fe521-dfc3-4afe-ba0c-1fd2022f7f35 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:00.180095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 541/1063] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:44:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:00.186684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c496187-43f7-4356-8b00-8b0133cb603b None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:00.187237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c496187-43f7-4356-8b00-8b0133cb603b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:00.187237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c496187-43f7-4356-8b00-8b0133cb603b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:00.263597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-4c496187-43f7-4356-8b00-8b0133cb603b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:00.264804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4c496187-43f7-4356-8b00-8b0133cb603b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:00.367992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 523/1064] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:44:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:00.659466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16fe521-dfc3-4afe-ba0c-1fd2022f7f35 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:00.662009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b16fe521-dfc3-4afe-ba0c-1fd2022f7f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:00.744280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:00.817730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16fe521-dfc3-4afe-ba0c-1fd2022f7f35 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:00.934394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1e2c2d-d4d4-4fb5-b398-14e1ce1caab8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:00.934911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1e2c2d-d4d4-4fb5-b398-14e1ce1caab8 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:00.935187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1e2c2d-d4d4-4fb5-b398-14e1ce1caab8 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:00.935456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1e2c2d-d4d4-4fb5-b398-14e1ce1caab8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:00.937062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-eb1e2c2d-d4d4-4fb5-b398-14e1ce1caab8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:00.980944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:01.035968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1e2c2d-d4d4-4fb5-b398-14e1ce1caab8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:01.037874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1e2c2d-d4d4-4fb5-b398-14e1ce1caab8 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:01.056587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 542/1065] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:01.842984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675dd3d-0e53-419a-b9d2-7737ab23a0d1 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:01.843550 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675dd3d-0e53-419a-b9d2-7737ab23a0d1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:01.843550 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675dd3d-0e53-419a-b9d2-7737ab23a0d1 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:01.844883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 524/1066] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:44:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:01.850481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeac3b8-79a3-412d-891b-ef644002b547 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:01.851331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeac3b8-79a3-412d-891b-ef644002b547 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:01.852326 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeac3b8-79a3-412d-891b-ef644002b547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:01.929067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ceeac3b8-79a3-412d-891b-ef644002b547 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:01.930306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ceeac3b8-79a3-412d-891b-ef644002b547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:02.062678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 543/1067] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:44:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:02.377302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9af103-fda0-4434-9459-b0b067f3f289 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:02.377580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9af103-fda0-4434-9459-b0b067f3f289 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:02.377854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9af103-fda0-4434-9459-b0b067f3f289 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:02.380025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 525/1068] 10.222.0.39 () {54 vars in 962 bytes} [Mon Oct 24 08:44:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:02.385737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46007894-bb23-4ab4-af43-bae70a5211ab None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:02.386022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46007894-bb23-4ab4-af43-bae70a5211ab None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:02.386296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46007894-bb23-4ab4-af43-bae70a5211ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:02.403674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:02.481328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-46007894-bb23-4ab4-af43-bae70a5211ab None None] MFA Rules not processed for user `5a4fd11d5d85479c9aa1d87b23a13141`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:02.482464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-46007894-bb23-4ab4-af43-bae70a5211ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:02.560358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 544/1069] 10.222.0.39 () {58 vars in 1059 bytes} [Mon Oct 24 08:44:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:02.568845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9af103-fda0-4434-9459-b0b067f3f289 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:02.570350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5e9af103-fda0-4434-9459-b0b067f3f289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:02.601676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:02.676054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9af103-fda0-4434-9459-b0b067f3f289 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:02.752262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02604f0-1818-48dd-8dd8-c2636f6d017c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a4fd11d5d85479c9aa1d87b23a13141', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:02.752774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02604f0-1818-48dd-8dd8-c2636f6d017c service glance] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:02.753051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02604f0-1818-48dd-8dd8-c2636f6d017c service glance] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:02.753360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02604f0-1818-48dd-8dd8-c2636f6d017c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:02.755498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a02604f0-1818-48dd-8dd8-c2636f6d017c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:02.840456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:02.874758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02604f0-1818-48dd-8dd8-c2636f6d017c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:02.876247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02604f0-1818-48dd-8dd8-c2636f6d017c service glance] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:02.894931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 526/1070] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:03.326431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b7d06b-0e63-477f-ab3b-05b68a3409d0 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:03.327155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b7d06b-0e63-477f-ab3b-05b68a3409d0 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:03.327155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b7d06b-0e63-477f-ab3b-05b68a3409d0 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:03.328367 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 545/1071] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:44:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:03.334899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc34f23-4c51-4333-adc8-ddc409ca47ab None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:03.335191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc34f23-4c51-4333-adc8-ddc409ca47ab None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:03.335464 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc34f23-4c51-4333-adc8-ddc409ca47ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:03.419521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-1bc34f23-4c51-4333-adc8-ddc409ca47ab None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:03.420834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1bc34f23-4c51-4333-adc8-ddc409ca47ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:03.538224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 527/1072] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:44:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:03.667665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4908c-675b-4b25-afe1-cfa54ca2364e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:03.668022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4908c-675b-4b25-afe1-cfa54ca2364e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:03.668392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4908c-675b-4b25-afe1-cfa54ca2364e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:03.736940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-46c4908c-675b-4b25-afe1-cfa54ca2364e None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:03.738495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-46c4908c-675b-4b25-afe1-cfa54ca2364e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:03.794745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:03.827120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 546/1073] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:44:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:03.841284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc34f23-4c51-4333-adc8-ddc409ca47ab None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:03.842590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1bc34f23-4c51-4333-adc8-ddc409ca47ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:03.871539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b7930d-90c2-4997-861a-1dcf669b929a None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:03.871818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b7930d-90c2-4997-861a-1dcf669b929a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:03.872091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b7930d-90c2-4997-861a-1dcf669b929a None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:03.873723 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 547/1074] 10.222.0.39 () {54 vars in 943 bytes} [Mon Oct 24 08:44:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:03.880517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d42890d-d69f-4dd1-a854-fb0a76fb36cb None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:03.880800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d42890d-d69f-4dd1-a854-fb0a76fb36cb None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:03.881072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d42890d-d69f-4dd1-a854-fb0a76fb36cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:03.887185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:03.961322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0d42890d-d69f-4dd1-a854-fb0a76fb36cb None None] MFA Rules not processed for user `60a07d37339f4f83ad2ce8be3d0892a8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:03.962581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0d42890d-d69f-4dd1-a854-fb0a76fb36cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:03.969218 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc34f23-4c51-4333-adc8-ddc409ca47ab None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:04.092580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 548/1075] 10.222.0.39 () {58 vars in 1040 bytes} [Mon Oct 24 08:44:03 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:04.105447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d42890d-d69f-4dd1-a854-fb0a76fb36cb None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:04.116064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d42890d-d69f-4dd1-a854-fb0a76fb36cb None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:04.120106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097e97b2-3f62-4909-85c8-3db214416bcb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:04.120829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097e97b2-3f62-4909-85c8-3db214416bcb service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:04.121113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097e97b2-3f62-4909-85c8-3db214416bcb service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:04.121390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097e97b2-3f62-4909-85c8-3db214416bcb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:04.123253 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-097e97b2-3f62-4909-85c8-3db214416bcb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:04.144796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:04.166476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:04.231680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097e97b2-3f62-4909-85c8-3db214416bcb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:04.233308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097e97b2-3f62-4909-85c8-3db214416bcb service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:04.240952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee29b1bd-2214-479a-9f10-27f28acebb5f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:04.241619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29b1bd-2214-479a-9f10-27f28acebb5f service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:04.241880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29b1bd-2214-479a-9f10-27f28acebb5f service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:04.242147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29b1bd-2214-479a-9f10-27f28acebb5f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:04.243910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ee29b1bd-2214-479a-9f10-27f28acebb5f service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:04.253833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 528/1076] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:04.337364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee29b1bd-2214-479a-9f10-27f28acebb5f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:04.339014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee29b1bd-2214-479a-9f10-27f28acebb5f service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:04.348691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 549/1077] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:44:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 24 08:44:06.655399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24ba31-3723-4284-9bd2-c5327b9bad91 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:06.655958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24ba31-3723-4284-9bd2-c5327b9bad91 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:06.655958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24ba31-3723-4284-9bd2-c5327b9bad91 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:06.662957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 529/1078] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:44:06 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:06.664420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f1c7bc-bb67-4a9e-a030-415ba4e990f0 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:06.664895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f1c7bc-bb67-4a9e-a030-415ba4e990f0 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:06.665015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f1c7bc-bb67-4a9e-a030-415ba4e990f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:06.746789 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-22f1c7bc-bb67-4a9e-a030-415ba4e990f0 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:06.748302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-22f1c7bc-bb67-4a9e-a030-415ba4e990f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:06.755574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:06.880159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 550/1079] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:44:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:44:07.164490 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24ba31-3723-4284-9bd2-c5327b9bad91 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:07.166378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2a24ba31-3723-4284-9bd2-c5327b9bad91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:07.335408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24ba31-3723-4284-9bd2-c5327b9bad91 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:07.348274 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:07.467464 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c542e-be9f-4d13-93db-0051b906b21e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:07.468059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c542e-be9f-4d13-93db-0051b906b21e service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:07.468453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c542e-be9f-4d13-93db-0051b906b21e service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:07.468732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c542e-be9f-4d13-93db-0051b906b21e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:07.470350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d17c542e-be9f-4d13-93db-0051b906b21e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:07.580577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17c542e-be9f-4d13-93db-0051b906b21e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:07.582341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17c542e-be9f-4d13-93db-0051b906b21e service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:07.604387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 530/1080] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:09.881899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01392fa7-f09e-4b64-934a-2514d169e7f2 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:09.882557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01392fa7-f09e-4b64-934a-2514d169e7f2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:09.882557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01392fa7-f09e-4b64-934a-2514d169e7f2 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:09.884001 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 551/1081] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:44:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:09.889441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d486b0f-c1f3-4219-8d7b-68a3f358c5cd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:09.889937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d486b0f-c1f3-4219-8d7b-68a3f358c5cd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:09.890142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d486b0f-c1f3-4219-8d7b-68a3f358c5cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:09.914002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:09.975078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2d486b0f-c1f3-4219-8d7b-68a3f358c5cd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:09.976357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2d486b0f-c1f3-4219-8d7b-68a3f358c5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:10.094446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 531/1082] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:44:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:10.389623 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01392fa7-f09e-4b64-934a-2514d169e7f2 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:10.391181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-01392fa7-f09e-4b64-934a-2514d169e7f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:10.567663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:10.580246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01392fa7-f09e-4b64-934a-2514d169e7f2 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:10.724660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4013959d-b882-432e-b477-723ad923458f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:10.725322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013959d-b882-432e-b477-723ad923458f service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:10.725603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013959d-b882-432e-b477-723ad923458f service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:10.725874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013959d-b882-432e-b477-723ad923458f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:10.727850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4013959d-b882-432e-b477-723ad923458f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:10.833239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4013959d-b882-432e-b477-723ad923458f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:10.834839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4013959d-b882-432e-b477-723ad923458f service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:10.845736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:10.857300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 552/1083] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:13.232339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7116334-786f-412e-ac36-3a7d84e4820d None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:13.232898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7116334-786f-412e-ac36-3a7d84e4820d None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:13.232898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7116334-786f-412e-ac36-3a7d84e4820d None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:13.235711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 532/1084] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:44:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:13.240111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf45b0e-4c47-45f0-aece-75890aeead9a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:13.240715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf45b0e-4c47-45f0-aece-75890aeead9a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:13.241011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf45b0e-4c47-45f0-aece-75890aeead9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:13.314402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-1bf45b0e-4c47-45f0-aece-75890aeead9a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:13.316274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1bf45b0e-4c47-45f0-aece-75890aeead9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:13.427983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 553/1085] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:44:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:44:13.736927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7116334-786f-412e-ac36-3a7d84e4820d None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:13.738693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b7116334-786f-412e-ac36-3a7d84e4820d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:13.846649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:13.911554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7116334-786f-412e-ac36-3a7d84e4820d None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:14.060228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16f07e5-1f8f-4ff4-bbfa-807a5fb9bdc4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:14.060833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f07e5-1f8f-4ff4-bbfa-807a5fb9bdc4 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:14.061109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f07e5-1f8f-4ff4-bbfa-807a5fb9bdc4 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:14.061380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f07e5-1f8f-4ff4-bbfa-807a5fb9bdc4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:14.063241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d16f07e5-1f8f-4ff4-bbfa-807a5fb9bdc4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:14.130658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:14.166708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16f07e5-1f8f-4ff4-bbfa-807a5fb9bdc4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:14.168327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16f07e5-1f8f-4ff4-bbfa-807a5fb9bdc4 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:14.192862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 533/1086] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 456 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:23.426420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=67 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:24.189600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=20 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:27.385561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a102751-c8bb-4181-b249-95adfcecac61 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:27.386137 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a102751-c8bb-4181-b249-95adfcecac61 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:27.386137 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a102751-c8bb-4181-b249-95adfcecac61 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:27.387456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 554/1087] 10.222.0.39 () {54 vars in 918 bytes} [Mon Oct 24 08:44:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:27.393000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8068b80-b044-4a56-8d2d-2be412b2231b None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:27.393927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8068b80-b044-4a56-8d2d-2be412b2231b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:27.394205 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8068b80-b044-4a56-8d2d-2be412b2231b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:27.476895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b8068b80-b044-4a56-8d2d-2be412b2231b None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:27.478134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b8068b80-b044-4a56-8d2d-2be412b2231b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:27.588144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 534/1088] 10.222.0.39 () {58 vars in 1015 bytes} [Mon Oct 24 08:44:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:44:27.604036 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd87fa-3e73-4bba-a316-b051890ed32a None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:27.604353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd87fa-3e73-4bba-a316-b051890ed32a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:27.604629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd87fa-3e73-4bba-a316-b051890ed32a None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:27.605970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 555/1089] 10.222.0.39 () {54 vars in 918 bytes} [Mon Oct 24 08:44:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:27.610668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaab649-25ac-4611-bae6-6718b332b5f5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:27.610959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaab649-25ac-4611-bae6-6718b332b5f5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:27.611257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaab649-25ac-4611-bae6-6718b332b5f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:27.695326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:27.698588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-faaab649-25ac-4611-bae6-6718b332b5f5 None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:27.700369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-faaab649-25ac-4611-bae6-6718b332b5f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:27.816245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 535/1090] 10.222.0.39 () {58 vars in 1015 bytes} [Mon Oct 24 08:44:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:30.452412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912200d9-b7ef-4d1e-a121-a047e3bc8d61 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:30.452986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912200d9-b7ef-4d1e-a121-a047e3bc8d61 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:30.452986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912200d9-b7ef-4d1e-a121-a047e3bc8d61 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:30.454302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 556/1091] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:44:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:30.460529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102bc57-06ef-4e49-9eaa-8078ea991c58 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:30.461081 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102bc57-06ef-4e49-9eaa-8078ea991c58 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:30.461206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102bc57-06ef-4e49-9eaa-8078ea991c58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:30.531244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2102bc57-06ef-4e49-9eaa-8078ea991c58 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:30.532509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2102bc57-06ef-4e49-9eaa-8078ea991c58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:30.591500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:30.639728 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 536/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:44:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:30.936311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912200d9-b7ef-4d1e-a121-a047e3bc8d61 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:30.937941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-912200d9-b7ef-4d1e-a121-a047e3bc8d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:31.102160 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912200d9-b7ef-4d1e-a121-a047e3bc8d61 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:31.178412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:31.220812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c7278f-c7e1-4491-af24-6bb219428c0a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:31.222021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c7278f-c7e1-4491-af24-6bb219428c0a service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:31.222298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c7278f-c7e1-4491-af24-6bb219428c0a service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:31.222569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c7278f-c7e1-4491-af24-6bb219428c0a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:31.224176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-21c7278f-c7e1-4491-af24-6bb219428c0a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:31.315724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c7278f-c7e1-4491-af24-6bb219428c0a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:31.317363 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c7278f-c7e1-4491-af24-6bb219428c0a service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:31.336013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 557/1093] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:32.502838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a29e5d0-5a6c-4f44-b971-cc5feed84957 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:32.503409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a29e5d0-5a6c-4f44-b971-cc5feed84957 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:32.503409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a29e5d0-5a6c-4f44-b971-cc5feed84957 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:32.504770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 537/1094] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:44:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:32.509896 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7fd4a3-44b5-4b8c-b8d1-ce639868f7c1 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:32.510360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7fd4a3-44b5-4b8c-b8d1-ce639868f7c1 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:32.510465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7fd4a3-44b5-4b8c-b8d1-ce639868f7c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:32.588714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ab7fd4a3-44b5-4b8c-b8d1-ce639868f7c1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:32.589920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ab7fd4a3-44b5-4b8c-b8d1-ce639868f7c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:32.618581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:32.691177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 558/1095] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:44:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:32.857541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a29e5d0-5a6c-4f44-b971-cc5feed84957 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:32.859085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-8a29e5d0-5a6c-4f44-b971-cc5feed84957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:33.016732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a29e5d0-5a6c-4f44-b971-cc5feed84957 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:33.059061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:33.135917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd7783f-f214-4090-9c46-1dc8661472c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:33.136622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd7783f-f214-4090-9c46-1dc8661472c6 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:33.136902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd7783f-f214-4090-9c46-1dc8661472c6 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:33.137176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd7783f-f214-4090-9c46-1dc8661472c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:33.138786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3bd7783f-f214-4090-9c46-1dc8661472c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:33.230503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd7783f-f214-4090-9c46-1dc8661472c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:33.232095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd7783f-f214-4090-9c46-1dc8661472c6 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:33.251610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 538/1096] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:34.412501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad21842-1212-4859-a9c0-fdaa9fcfa8b5 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:34.413091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad21842-1212-4859-a9c0-fdaa9fcfa8b5 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:34.413091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad21842-1212-4859-a9c0-fdaa9fcfa8b5 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:34.414402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 559/1097] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:44:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:34.419505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd1e5d6-e393-47d1-8037-0f5b68eed64f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:34.419979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd1e5d6-e393-47d1-8037-0f5b68eed64f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:34.420085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd1e5d6-e393-47d1-8037-0f5b68eed64f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:34.474712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:34.498269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-dfd1e5d6-e393-47d1-8037-0f5b68eed64f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:34.499495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-dfd1e5d6-e393-47d1-8037-0f5b68eed64f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:34.598089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 539/1098] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:44:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:34.756278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad21842-1212-4859-a9c0-fdaa9fcfa8b5 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:34.757744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-aad21842-1212-4859-a9c0-fdaa9fcfa8b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:34.916834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad21842-1212-4859-a9c0-fdaa9fcfa8b5 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:34.929729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:35.039193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9f4480-6e9e-48a9-ae2c-fbe6bd89727c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:35.039712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f4480-6e9e-48a9-ae2c-fbe6bd89727c service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:35.039992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f4480-6e9e-48a9-ae2c-fbe6bd89727c service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:35.040319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f4480-6e9e-48a9-ae2c-fbe6bd89727c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:35.041916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bc9f4480-6e9e-48a9-ae2c-fbe6bd89727c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:35.135043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9f4480-6e9e-48a9-ae2c-fbe6bd89727c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:35.136670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9f4480-6e9e-48a9-ae2c-fbe6bd89727c service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:35.155195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 560/1099] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:36.300702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8eafbb-17af-4219-8c54-d0f9cfa57b99 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:36.301237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8eafbb-17af-4219-8c54-d0f9cfa57b99 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:36.301237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8eafbb-17af-4219-8c54-d0f9cfa57b99 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:36.302535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 540/1100] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:44:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:36.307925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3943bc1-449d-46c8-9f63-43680398f4b5 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:36.308452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3943bc1-449d-46c8-9f63-43680398f4b5 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:36.308553 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3943bc1-449d-46c8-9f63-43680398f4b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:36.328324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:36.387220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f3943bc1-449d-46c8-9f63-43680398f4b5 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:36.388447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f3943bc1-449d-46c8-9f63-43680398f4b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:36.487572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 561/1101] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:44:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:36.641903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8eafbb-17af-4219-8c54-d0f9cfa57b99 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:36.643428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1f8eafbb-17af-4219-8c54-d0f9cfa57b99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:36.760002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:36.796619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8eafbb-17af-4219-8c54-d0f9cfa57b99 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:36.921310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99773a56-ec88-4106-9466-486af9edbbff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:36.921863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99773a56-ec88-4106-9466-486af9edbbff service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:36.922144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99773a56-ec88-4106-9466-486af9edbbff service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:36.922418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99773a56-ec88-4106-9466-486af9edbbff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:36.924677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-99773a56-ec88-4106-9466-486af9edbbff service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:37.013427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:37.027161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99773a56-ec88-4106-9466-486af9edbbff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:37.028790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99773a56-ec88-4106-9466-486af9edbbff service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:37.047446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 541/1102] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:38.186915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d3c349-a1cc-414f-95ff-cd45207caa85 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:38.187492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d3c349-a1cc-414f-95ff-cd45207caa85 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:38.187492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d3c349-a1cc-414f-95ff-cd45207caa85 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:38.188857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 562/1103] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:44:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:38.194079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c42c78-6bd3-47a7-9eb2-e7b7f69697fb None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:38.194567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c42c78-6bd3-47a7-9eb2-e7b7f69697fb None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:38.194673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c42c78-6bd3-47a7-9eb2-e7b7f69697fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:38.275043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-22c42c78-6bd3-47a7-9eb2-e7b7f69697fb None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:38.276192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-22c42c78-6bd3-47a7-9eb2-e7b7f69697fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:38.375138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 542/1104] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:44:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:38.529465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d3c349-a1cc-414f-95ff-cd45207caa85 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:38.530971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b3d3c349-a1cc-414f-95ff-cd45207caa85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:38.621555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:38.685295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d3c349-a1cc-414f-95ff-cd45207caa85 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:38.802343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb5db8c-9aba-4c05-bda3-d2679a5eea30 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:38.802868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb5db8c-9aba-4c05-bda3-d2679a5eea30 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:38.803144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb5db8c-9aba-4c05-bda3-d2679a5eea30 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:38.803411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb5db8c-9aba-4c05-bda3-d2679a5eea30 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:38.805028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bcb5db8c-9aba-4c05-bda3-d2679a5eea30 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:38.857352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:38.900416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb5db8c-9aba-4c05-bda3-d2679a5eea30 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:38.902022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb5db8c-9aba-4c05-bda3-d2679a5eea30 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:38.920801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 563/1105] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:39.989088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe64ed-f695-4dc1-92b6-d162ede2fe90 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:39.989733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe64ed-f695-4dc1-92b6-d162ede2fe90 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:39.989733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe64ed-f695-4dc1-92b6-d162ede2fe90 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:39.991136 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 543/1106] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:44:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:39.996315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d66694-d401-4634-aedb-8ab166be50d3 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:39.996776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d66694-d401-4634-aedb-8ab166be50d3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:39.996877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d66694-d401-4634-aedb-8ab166be50d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:40.075850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d1d66694-d401-4634-aedb-8ab166be50d3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:40.077103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d1d66694-d401-4634-aedb-8ab166be50d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:40.174484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 564/1107] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:44:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:40.326432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe64ed-f695-4dc1-92b6-d162ede2fe90 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:40.327917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-71fe64ed-f695-4dc1-92b6-d162ede2fe90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:40.379433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:40.479176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe64ed-f695-4dc1-92b6-d162ede2fe90 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:40.592837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53f6429-8c50-4579-a5bc-010a76af6644 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:40.593441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53f6429-8c50-4579-a5bc-010a76af6644 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:40.593723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53f6429-8c50-4579-a5bc-010a76af6644 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:40.593999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53f6429-8c50-4579-a5bc-010a76af6644 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:40.596349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b53f6429-8c50-4579-a5bc-010a76af6644 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:40.603413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:40.687561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53f6429-8c50-4579-a5bc-010a76af6644 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:40.689255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53f6429-8c50-4579-a5bc-010a76af6644 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:40.708163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 544/1108] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:41.773987 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95659676-5bde-4789-a835-43b1bf935627 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:41.774564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95659676-5bde-4789-a835-43b1bf935627 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:41.774564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95659676-5bde-4789-a835-43b1bf935627 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:41.775866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 565/1109] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:44:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:41.780878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25ee0c3-848e-40ce-91c3-64cdf14a358c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:41.781313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25ee0c3-848e-40ce-91c3-64cdf14a358c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:41.781410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25ee0c3-848e-40ce-91c3-64cdf14a358c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:41.859050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-d25ee0c3-848e-40ce-91c3-64cdf14a358c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:41.860283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d25ee0c3-848e-40ce-91c3-64cdf14a358c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:41.928972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:41.958168 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 545/1110] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:44:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:42.110496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95659676-5bde-4789-a835-43b1bf935627 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:42.112659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-95659676-5bde-4789-a835-43b1bf935627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:42.130623 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:42.266556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95659676-5bde-4789-a835-43b1bf935627 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:42.363523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:42.393071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7156e5e-9195-40d6-b4b6-4ff2f1591b2f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:42.393737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7156e5e-9195-40d6-b4b6-4ff2f1591b2f service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:42.394006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7156e5e-9195-40d6-b4b6-4ff2f1591b2f service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:42.394270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7156e5e-9195-40d6-b4b6-4ff2f1591b2f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:42.395828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f7156e5e-9195-40d6-b4b6-4ff2f1591b2f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:42.487007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7156e5e-9195-40d6-b4b6-4ff2f1591b2f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:42.488620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7156e5e-9195-40d6-b4b6-4ff2f1591b2f service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:42.506956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 566/1111] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:43.574263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b60b11-3b2f-49b9-be28-67f57bf919b6 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:43.574870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b60b11-3b2f-49b9-be28-67f57bf919b6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:43.575065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b60b11-3b2f-49b9-be28-67f57bf919b6 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:43.576543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 546/1112] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:44:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:43.581713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ab361-1ae5-4ac5-ba4b-53c708ad8364 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:43.582171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ab361-1ae5-4ac5-ba4b-53c708ad8364 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:43.582267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ab361-1ae5-4ac5-ba4b-53c708ad8364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:43.660107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-919ab361-1ae5-4ac5-ba4b-53c708ad8364 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:43.661329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-919ab361-1ae5-4ac5-ba4b-53c708ad8364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:43.702055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:43.761220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 567/1113] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:44:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:43.922402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b60b11-3b2f-49b9-be28-67f57bf919b6 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:43.924091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-19b60b11-3b2f-49b9-be28-67f57bf919b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:44.092781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b60b11-3b2f-49b9-be28-67f57bf919b6 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:44.149626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:44.209962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009cb97c-9206-4d8f-9f0e-8ac5eef48737 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:44.210476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009cb97c-9206-4d8f-9f0e-8ac5eef48737 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:44.210763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009cb97c-9206-4d8f-9f0e-8ac5eef48737 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:44.211038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009cb97c-9206-4d8f-9f0e-8ac5eef48737 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:44.212672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-009cb97c-9206-4d8f-9f0e-8ac5eef48737 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:44.303450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009cb97c-9206-4d8f-9f0e-8ac5eef48737 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:44.305068 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009cb97c-9206-4d8f-9f0e-8ac5eef48737 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:44.324014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 547/1114] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:45.387328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce7010-d1f3-485c-b6db-3d8f12a62028 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:45.387952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce7010-d1f3-485c-b6db-3d8f12a62028 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:45.387952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce7010-d1f3-485c-b6db-3d8f12a62028 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:45.389265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 568/1115] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:44:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:45.394178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc4ac2b-986f-477c-9eb4-365ea615dedd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:45.394644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc4ac2b-986f-477c-9eb4-365ea615dedd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:45.394744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc4ac2b-986f-477c-9eb4-365ea615dedd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:45.473061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5bc4ac2b-986f-477c-9eb4-365ea615dedd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:45.474259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5bc4ac2b-986f-477c-9eb4-365ea615dedd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:45.476298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:45.572930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 548/1116] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:44:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:45.728082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ce7010-d1f3-485c-b6db-3d8f12a62028 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:45.729537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a3ce7010-d1f3-485c-b6db-3d8f12a62028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:45.878476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ce7010-d1f3-485c-b6db-3d8f12a62028 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:45.906370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:45.998955 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2683f79-c2c6-43f1-bccf-c58e49cd6205 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:45.999560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2683f79-c2c6-43f1-bccf-c58e49cd6205 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:45.999836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2683f79-c2c6-43f1-bccf-c58e49cd6205 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:46.000106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2683f79-c2c6-43f1-bccf-c58e49cd6205 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:46.001701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e2683f79-c2c6-43f1-bccf-c58e49cd6205 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:46.096230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2683f79-c2c6-43f1-bccf-c58e49cd6205 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:46.097911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2683f79-c2c6-43f1-bccf-c58e49cd6205 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:46.117091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 569/1117] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:48.222831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231b3d0-3aad-4423-acc6-2b26a664518f None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:48.223428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231b3d0-3aad-4423-acc6-2b26a664518f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:48.223428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231b3d0-3aad-4423-acc6-2b26a664518f None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:48.224739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 549/1118] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:44:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:48.230329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73c5c94-5c0c-4761-aabe-b1957444da50 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:48.230763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73c5c94-5c0c-4761-aabe-b1957444da50 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:48.230881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73c5c94-5c0c-4761-aabe-b1957444da50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:48.272709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:48.299510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b73c5c94-5c0c-4761-aabe-b1957444da50 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:48.300753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b73c5c94-5c0c-4761-aabe-b1957444da50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:48.397688 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 570/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:44:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:44:48.683147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4231b3d0-3aad-4423-acc6-2b26a664518f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:48.684717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4231b3d0-3aad-4423-acc6-2b26a664518f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:48.790846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4231b3d0-3aad-4423-acc6-2b26a664518f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:48.823709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:48.869806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cd8470-f80b-4f8e-b9a7-e78ce53e9f97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:48.870407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd8470-f80b-4f8e-b9a7-e78ce53e9f97 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:48.870690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd8470-f80b-4f8e-b9a7-e78ce53e9f97 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:48.870962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd8470-f80b-4f8e-b9a7-e78ce53e9f97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:48.872866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-87cd8470-f80b-4f8e-b9a7-e78ce53e9f97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:48.987493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cd8470-f80b-4f8e-b9a7-e78ce53e9f97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:48.988963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cd8470-f80b-4f8e-b9a7-e78ce53e9f97 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:49.014027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 550/1120] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:51.410390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb2c3a2-75fc-46f1-b047-348ccea80a3d None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:51.410978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb2c3a2-75fc-46f1-b047-348ccea80a3d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:51.411090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb2c3a2-75fc-46f1-b047-348ccea80a3d None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:51.412472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 571/1121] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:44:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:51.418199 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccab33-c791-42a3-b30c-f0c414886278 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:51.418659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccab33-c791-42a3-b30c-f0c414886278 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:51.418782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccab33-c791-42a3-b30c-f0c414886278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:51.490869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5dccab33-c791-42a3-b30c-f0c414886278 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:51.492085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5dccab33-c791-42a3-b30c-f0c414886278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:51.506735 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:51.595573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 551/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:44:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:51.883677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb2c3a2-75fc-46f1-b047-348ccea80a3d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:51.885310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-afb2c3a2-75fc-46f1-b047-348ccea80a3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:51.993653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb2c3a2-75fc-46f1-b047-348ccea80a3d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:52.008428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:52.074497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c295036-8c35-4f5c-9996-877968357205 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:52.075113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c295036-8c35-4f5c-9996-877968357205 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:52.075392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c295036-8c35-4f5c-9996-877968357205 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:52.075663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c295036-8c35-4f5c-9996-877968357205 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:52.077285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8c295036-8c35-4f5c-9996-877968357205 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:52.195172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c295036-8c35-4f5c-9996-877968357205 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:52.196567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c295036-8c35-4f5c-9996-877968357205 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:52.218752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 572/1123] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:54.757190 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b219c6e-5a7f-42d9-b2b9-ea10d5f11841 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:54.757759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b219c6e-5a7f-42d9-b2b9-ea10d5f11841 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:54.757759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b219c6e-5a7f-42d9-b2b9-ea10d5f11841 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:54.759149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 552/1124] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:44:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:54.764680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de793a8b-cf36-477a-b083-5b471428f503 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:54.765087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de793a8b-cf36-477a-b083-5b471428f503 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:54.765234 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de793a8b-cf36-477a-b083-5b471428f503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:54.812374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:54.832427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-de793a8b-cf36-477a-b083-5b471428f503 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:54.833625 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-de793a8b-cf36-477a-b083-5b471428f503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:54.932009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 573/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:44:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:55.283779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b219c6e-5a7f-42d9-b2b9-ea10d5f11841 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:55.285350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9b219c6e-5a7f-42d9-b2b9-ea10d5f11841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:55.406647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b219c6e-5a7f-42d9-b2b9-ea10d5f11841 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:55.448691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:55.492056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44800ccb-09da-4968-a74c-c26897f36f61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:55.492694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44800ccb-09da-4968-a74c-c26897f36f61 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:55.492967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44800ccb-09da-4968-a74c-c26897f36f61 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:55.493235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44800ccb-09da-4968-a74c-c26897f36f61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:55.494807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-44800ccb-09da-4968-a74c-c26897f36f61 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:55.607131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44800ccb-09da-4968-a74c-c26897f36f61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:55.608535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44800ccb-09da-4968-a74c-c26897f36f61 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:55.627660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 553/1126] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:44:58.031274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc6b72-8e81-412f-8781-893d9a00b930 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:58.031780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc6b72-8e81-412f-8781-893d9a00b930 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:58.031920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc6b72-8e81-412f-8781-893d9a00b930 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:58.033423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 574/1127] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:44:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:44:58.039037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d75f26c-1fad-4ac3-8bfc-6062dabe04ec None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:58.039506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d75f26c-1fad-4ac3-8bfc-6062dabe04ec None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:58.039609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d75f26c-1fad-4ac3-8bfc-6062dabe04ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:58.104965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5d75f26c-1fad-4ac3-8bfc-6062dabe04ec None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:44:58.106166 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5d75f26c-1fad-4ac3-8bfc-6062dabe04ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:58.128194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:58.204793 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 554/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:44:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:44:58.531403 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc6b72-8e81-412f-8781-893d9a00b930 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:44:58.535136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-83dc6b72-8e81-412f-8781-893d9a00b930 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:58.668858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc6b72-8e81-412f-8781-893d9a00b930 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:44:58.694023 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:44:58.750319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900a6fe4-2449-4513-8e1e-0c01a3cbbc85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:44:58.751580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a6fe4-2449-4513-8e1e-0c01a3cbbc85 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:44:58.751857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a6fe4-2449-4513-8e1e-0c01a3cbbc85 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:44:58.752152 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a6fe4-2449-4513-8e1e-0c01a3cbbc85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:44:58.753751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-900a6fe4-2449-4513-8e1e-0c01a3cbbc85 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:44:58.876998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900a6fe4-2449-4513-8e1e-0c01a3cbbc85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:44:58.878416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900a6fe4-2449-4513-8e1e-0c01a3cbbc85 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:44:58.900766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 575/1129] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:44:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:01.303346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc349c6-6dbe-4d4e-b3dd-05227e755cc5 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:01.303881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc349c6-6dbe-4d4e-b3dd-05227e755cc5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:01.303881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc349c6-6dbe-4d4e-b3dd-05227e755cc5 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:01.305281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 555/1130] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:45:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:01.310977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99533e3-a0a9-42a4-bc07-fc81abae0d0d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:01.311417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99533e3-a0a9-42a4-bc07-fc81abae0d0d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:01.311519 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99533e3-a0a9-42a4-bc07-fc81abae0d0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:01.384898 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:01.388280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c99533e3-a0a9-42a4-bc07-fc81abae0d0d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:01.390017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c99533e3-a0a9-42a4-bc07-fc81abae0d0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:01.500946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 576/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:45:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:01.836268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc349c6-6dbe-4d4e-b3dd-05227e755cc5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:01.837949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-afc349c6-6dbe-4d4e-b3dd-05227e755cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:01.951034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc349c6-6dbe-4d4e-b3dd-05227e755cc5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:02.007765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:02.035301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018d4c5-2b8c-4eae-815c-96e71a70cd02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:02.035902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018d4c5-2b8c-4eae-815c-96e71a70cd02 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:02.036206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018d4c5-2b8c-4eae-815c-96e71a70cd02 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:02.036479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018d4c5-2b8c-4eae-815c-96e71a70cd02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:02.038069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5018d4c5-2b8c-4eae-815c-96e71a70cd02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:02.149643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5018d4c5-2b8c-4eae-815c-96e71a70cd02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:02.151132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5018d4c5-2b8c-4eae-815c-96e71a70cd02 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:02.169338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 556/1132] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:04.511509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f2a7c-6b6a-4dd6-9c07-52c30a117bf9 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:04.512079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f2a7c-6b6a-4dd6-9c07-52c30a117bf9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:04.512079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f2a7c-6b6a-4dd6-9c07-52c30a117bf9 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:04.513507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 577/1133] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:45:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:04.519056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25859aa-0623-45ae-9456-e55b54dd73dd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:04.519498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25859aa-0623-45ae-9456-e55b54dd73dd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:04.519680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25859aa-0623-45ae-9456-e55b54dd73dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:04.586343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-d25859aa-0623-45ae-9456-e55b54dd73dd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:04.587564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d25859aa-0623-45ae-9456-e55b54dd73dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:04.617091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:04.686977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 557/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:45:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:04.976581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f2a7c-6b6a-4dd6-9c07-52c30a117bf9 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:04.978146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-618f2a7c-6b6a-4dd6-9c07-52c30a117bf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:05.086435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f2a7c-6b6a-4dd6-9c07-52c30a117bf9 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:05.114765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:05.163323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046ea2d3-aeae-464f-9127-2b0b1203ab70 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:05.163933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046ea2d3-aeae-464f-9127-2b0b1203ab70 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:05.164249 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046ea2d3-aeae-464f-9127-2b0b1203ab70 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:05.164525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046ea2d3-aeae-464f-9127-2b0b1203ab70 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:05.166136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-046ea2d3-aeae-464f-9127-2b0b1203ab70 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:05.277706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046ea2d3-aeae-464f-9127-2b0b1203ab70 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:05.279126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046ea2d3-aeae-464f-9127-2b0b1203ab70 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:05.297547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 578/1135] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:07.553995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131f9cb9-3cd8-4c44-be40-de09a1c4d042 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:07.554535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131f9cb9-3cd8-4c44-be40-de09a1c4d042 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:07.554535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131f9cb9-3cd8-4c44-be40-de09a1c4d042 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:07.555895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 558/1136] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:45:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:07.561479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c61b6-2dd8-4057-ba42-110a2b28efd5 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:07.561867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c61b6-2dd8-4057-ba42-110a2b28efd5 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:07.562027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c61b6-2dd8-4057-ba42-110a2b28efd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:07.627567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-442c61b6-2dd8-4057-ba42-110a2b28efd5 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:07.628812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-442c61b6-2dd8-4057-ba42-110a2b28efd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:07.637466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:07.725688 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 579/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:45:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:08.015352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131f9cb9-3cd8-4c44-be40-de09a1c4d042 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:08.017758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-131f9cb9-3cd8-4c44-be40-de09a1c4d042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:08.119810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131f9cb9-3cd8-4c44-be40-de09a1c4d042 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:08.175127 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:08.193264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca80ab9-92f7-45bd-a0aa-3f388e2955cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:08.193861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca80ab9-92f7-45bd-a0aa-3f388e2955cb service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:08.194137 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca80ab9-92f7-45bd-a0aa-3f388e2955cb service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:08.194403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca80ab9-92f7-45bd-a0aa-3f388e2955cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:08.195961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bca80ab9-92f7-45bd-a0aa-3f388e2955cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:08.304225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca80ab9-92f7-45bd-a0aa-3f388e2955cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:08.305032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca80ab9-92f7-45bd-a0aa-3f388e2955cb service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:08.322656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 559/1138] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:10.578972 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b096ef3-ccc2-4664-ac5d-c79a80ad8943 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:10.579542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b096ef3-ccc2-4664-ac5d-c79a80ad8943 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:10.579542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b096ef3-ccc2-4664-ac5d-c79a80ad8943 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:10.580922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 580/1139] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:45:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:10.586544 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7b831c-4d79-4946-a94f-b4cee5f0c39b None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:10.586941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7b831c-4d79-4946-a94f-b4cee5f0c39b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:10.587097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7b831c-4d79-4946-a94f-b4cee5f0c39b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:10.652982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5f7b831c-4d79-4946-a94f-b4cee5f0c39b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:10.654182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5f7b831c-4d79-4946-a94f-b4cee5f0c39b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:10.692994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:10.751549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 560/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:45:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:11.035807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b096ef3-ccc2-4664-ac5d-c79a80ad8943 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:11.037359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9b096ef3-ccc2-4664-ac5d-c79a80ad8943 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:11.142770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b096ef3-ccc2-4664-ac5d-c79a80ad8943 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:11.180950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:11.217388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c0a31a-a1d5-445c-8f91-a0ef42d5f8d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:11.218004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c0a31a-a1d5-445c-8f91-a0ef42d5f8d8 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:11.219209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c0a31a-a1d5-445c-8f91-a0ef42d5f8d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:11.219488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c0a31a-a1d5-445c-8f91-a0ef42d5f8d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:11.221142 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c5c0a31a-a1d5-445c-8f91-a0ef42d5f8d8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:11.331760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c0a31a-a1d5-445c-8f91-a0ef42d5f8d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:11.333235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c0a31a-a1d5-445c-8f91-a0ef42d5f8d8 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:11.351447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 581/1141] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:13.542147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04abcec-b429-4914-a7aa-9135a0761f37 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:13.542779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04abcec-b429-4914-a7aa-9135a0761f37 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:13.542779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04abcec-b429-4914-a7aa-9135a0761f37 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:13.544029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 561/1142] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:45:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:13.549408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538d68d-1220-45a1-b6ff-be269d45359d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:13.549888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538d68d-1220-45a1-b6ff-be269d45359d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:13.549995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538d68d-1220-45a1-b6ff-be269d45359d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:13.616674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-1538d68d-1220-45a1-b6ff-be269d45359d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:13.617851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1538d68d-1220-45a1-b6ff-be269d45359d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:13.637400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:13.719397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 582/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:45:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:14.007764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04abcec-b429-4914-a7aa-9135a0761f37 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:14.009417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a04abcec-b429-4914-a7aa-9135a0761f37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:14.115203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04abcec-b429-4914-a7aa-9135a0761f37 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:14.182388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:14.190361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1cf5dc-05ce-42cf-9455-cc2e42e9c714 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:14.190963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1cf5dc-05ce-42cf-9455-cc2e42e9c714 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:14.191243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1cf5dc-05ce-42cf-9455-cc2e42e9c714 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:14.191514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1cf5dc-05ce-42cf-9455-cc2e42e9c714 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:14.193136 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3f1cf5dc-05ce-42cf-9455-cc2e42e9c714 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:14.302082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1cf5dc-05ce-42cf-9455-cc2e42e9c714 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:14.303426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1cf5dc-05ce-42cf-9455-cc2e42e9c714 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:14.321759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 562/1144] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:16.514982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5be5a6-dd96-4244-a3af-bbba930d1b7a None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:16.515731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5be5a6-dd96-4244-a3af-bbba930d1b7a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:16.515731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5be5a6-dd96-4244-a3af-bbba930d1b7a None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:16.516985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 583/1145] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:45:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:16.522589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8cc027-5bbe-431e-b832-8e2eae8e2067 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:16.523030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8cc027-5bbe-431e-b832-8e2eae8e2067 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:16.523140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8cc027-5bbe-431e-b832-8e2eae8e2067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:16.590251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7c8cc027-5bbe-431e-b832-8e2eae8e2067 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:16.591473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7c8cc027-5bbe-431e-b832-8e2eae8e2067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:16.642119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:16.691840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 563/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:45:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:16.992754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5be5a6-dd96-4244-a3af-bbba930d1b7a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:16.994501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3a5be5a6-dd96-4244-a3af-bbba930d1b7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:17.100489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5be5a6-dd96-4244-a3af-bbba930d1b7a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:17.151354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:17.177763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b07198-db51-499e-9d0c-4303a31d03c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:17.178370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b07198-db51-499e-9d0c-4303a31d03c1 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:17.178652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b07198-db51-499e-9d0c-4303a31d03c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:17.178924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b07198-db51-499e-9d0c-4303a31d03c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:17.180588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-22b07198-db51-499e-9d0c-4303a31d03c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:17.291476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b07198-db51-499e-9d0c-4303a31d03c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:17.294305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b07198-db51-499e-9d0c-4303a31d03c1 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:17.318432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 584/1147] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:26.690282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=22 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:27.316297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=66 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:38.018135 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae27282-fb1c-4dee-aa84-67fbce33e300 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:38.018686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae27282-fb1c-4dee-aa84-67fbce33e300 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:38.018686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae27282-fb1c-4dee-aa84-67fbce33e300 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:38.020003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 564/1148] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:45:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:38.025298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f053d2d-e480-431e-84d7-9a601bf6fd40 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:38.025731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f053d2d-e480-431e-84d7-9a601bf6fd40 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:38.025834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f053d2d-e480-431e-84d7-9a601bf6fd40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:38.094281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-8f053d2d-e480-431e-84d7-9a601bf6fd40 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:38.095492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8f053d2d-e480-431e-84d7-9a601bf6fd40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:38.191518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 585/1149] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:45:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:38.493378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225f9f51-c4c4-45f3-be0d-e05db48834fe None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:38.493670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225f9f51-c4c4-45f3-be0d-e05db48834fe None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:38.493954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225f9f51-c4c4-45f3-be0d-e05db48834fe None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:38.495283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 565/1150] 10.222.0.39 () {54 vars in 962 bytes} [Mon Oct 24 08:45:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:38.500725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b28c37e-3946-4e5c-870d-114947daf497 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:38.501015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b28c37e-3946-4e5c-870d-114947daf497 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:38.501288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b28c37e-3946-4e5c-870d-114947daf497 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:38.577966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5b28c37e-3946-4e5c-870d-114947daf497 None None] MFA Rules not processed for user `5a4fd11d5d85479c9aa1d87b23a13141`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:38.579173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5b28c37e-3946-4e5c-870d-114947daf497 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:38.600238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:38.657797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 586/1151] 10.222.0.39 () {58 vars in 1059 bytes} [Mon Oct 24 08:45:38 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:38.665561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225f9f51-c4c4-45f3-be0d-e05db48834fe None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:38.667062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-225f9f51-c4c4-45f3-be0d-e05db48834fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:38.770638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225f9f51-c4c4-45f3-be0d-e05db48834fe None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:38.846039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b24b92-4673-4392-8939-cbfc83853f5e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a4fd11d5d85479c9aa1d87b23a13141', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:38.846646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b24b92-4673-4392-8939-cbfc83853f5e service glance] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:38.846906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b24b92-4673-4392-8939-cbfc83853f5e service glance] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:38.847179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b24b92-4673-4392-8939-cbfc83853f5e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:38.848787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-02b24b92-4673-4392-8939-cbfc83853f5e service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:38.903635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:38.958043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b24b92-4673-4392-8939-cbfc83853f5e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:38.959407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b24b92-4673-4392-8939-cbfc83853f5e service glance] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:38.978400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 566/1152] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:45:41.247091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96695a76-2ca8-4817-a1aa-50e04d3976ce None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:41.247685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96695a76-2ca8-4817-a1aa-50e04d3976ce None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:41.247685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96695a76-2ca8-4817-a1aa-50e04d3976ce None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:41.249526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 587/1153] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:45:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:41.254849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d4c748-76be-43b1-bdbc-85ef44dc6f67 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:41.255289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d4c748-76be-43b1-bdbc-85ef44dc6f67 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:41.255389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d4c748-76be-43b1-bdbc-85ef44dc6f67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:41.321929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-78d4c748-76be-43b1-bdbc-85ef44dc6f67 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:41.323136 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-78d4c748-76be-43b1-bdbc-85ef44dc6f67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:41.419758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 567/1154] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:45:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:41.695522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96695a76-2ca8-4817-a1aa-50e04d3976ce None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:41.697100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-96695a76-2ca8-4817-a1aa-50e04d3976ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:41.821284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96695a76-2ca8-4817-a1aa-50e04d3976ce None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:41.872711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:41.917170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0ce3ee-720b-4f7b-971e-852e32ccadcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:41.917770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ce3ee-720b-4f7b-971e-852e32ccadcb admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:41.918047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ce3ee-720b-4f7b-971e-852e32ccadcb admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:41.918316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ce3ee-720b-4f7b-971e-852e32ccadcb admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:41.918944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0ce3ee-720b-4f7b-971e-852e32ccadcb admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:41.920575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-0c0ce3ee-720b-4f7b-971e-852e32ccadcb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:45:41.920575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:45:41.920575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:45:41.921715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0ce3ee-720b-4f7b-971e-852e32ccadcb admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:41.941148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 588/1155] 10.222.0.39 () {56 vars in 1108 bytes} [Mon Oct 24 08:45:41 2022] GET /identity/v3/projects => generated 2553 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:45:44.135496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bcf7b-8046-4a66-aea3-c707573fd552 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:44.136026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bcf7b-8046-4a66-aea3-c707573fd552 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:44.136026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bcf7b-8046-4a66-aea3-c707573fd552 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:44.137415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 568/1156] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:45:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:44.143307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0978d4-f8a0-4553-9d14-3d380254ac98 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:44.143720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0978d4-f8a0-4553-9d14-3d380254ac98 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:44.143860 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0978d4-f8a0-4553-9d14-3d380254ac98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:44.210226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fc0978d4-f8a0-4553-9d14-3d380254ac98 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:44.211435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fc0978d4-f8a0-4553-9d14-3d380254ac98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:44.306534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 589/1157] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:44.613691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917bcf7b-8046-4a66-aea3-c707573fd552 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:44.615322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-917bcf7b-8046-4a66-aea3-c707573fd552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:44.625751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:44.768157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917bcf7b-8046-4a66-aea3-c707573fd552 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:44.848832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:44.883655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dba44e7-5f5a-4623-8a4e-d21ff4f1e8a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:44.884306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dba44e7-5f5a-4623-8a4e-d21ff4f1e8a0 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:44.884590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dba44e7-5f5a-4623-8a4e-d21ff4f1e8a0 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:44.884865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dba44e7-5f5a-4623-8a4e-d21ff4f1e8a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:44.886466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7dba44e7-5f5a-4623-8a4e-d21ff4f1e8a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:44.977734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dba44e7-5f5a-4623-8a4e-d21ff4f1e8a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:44.979337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dba44e7-5f5a-4623-8a4e-d21ff4f1e8a0 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:44.999367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 569/1158] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:47.397613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2124e3-2cd2-4218-b1fd-7b9a73755bb7 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:47.398148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2124e3-2cd2-4218-b1fd-7b9a73755bb7 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:47.398148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2124e3-2cd2-4218-b1fd-7b9a73755bb7 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:47.399515 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 590/1159] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:45:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:47.405121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f08ec-b509-46d8-9ec4-1bc7546c5fae None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:47.405551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f08ec-b509-46d8-9ec4-1bc7546c5fae None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:47.405678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f08ec-b509-46d8-9ec4-1bc7546c5fae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:47.473133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-584f08ec-b509-46d8-9ec4-1bc7546c5fae None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:47.474342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-584f08ec-b509-46d8-9ec4-1bc7546c5fae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:47.523466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:47.574742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 570/1160] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:45:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:47.878333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2124e3-2cd2-4218-b1fd-7b9a73755bb7 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:47.879837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ce2124e3-2cd2-4218-b1fd-7b9a73755bb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:47.903862 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:47.984770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2124e3-2cd2-4218-b1fd-7b9a73755bb7 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:48.059306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94320dc-68d3-4a82-8510-2484055fa1b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:48.059949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94320dc-68d3-4a82-8510-2484055fa1b0 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:48.060751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94320dc-68d3-4a82-8510-2484055fa1b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:48.061105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94320dc-68d3-4a82-8510-2484055fa1b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:48.062621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c94320dc-68d3-4a82-8510-2484055fa1b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:48.139293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:48.174669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94320dc-68d3-4a82-8510-2484055fa1b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:48.176098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94320dc-68d3-4a82-8510-2484055fa1b0 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:48.194918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 591/1161] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:50.459998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc473ac2-b9c4-45e7-8ac7-571e06fbf0b7 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:50.460576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc473ac2-b9c4-45e7-8ac7-571e06fbf0b7 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:50.460576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc473ac2-b9c4-45e7-8ac7-571e06fbf0b7 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:50.461935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 571/1162] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:45:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:50.467844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ee9fc1-0129-456d-acd6-8479f7c3f338 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:50.468328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ee9fc1-0129-456d-acd6-8479f7c3f338 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:50.468427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ee9fc1-0129-456d-acd6-8479f7c3f338 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:50.534979 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-80ee9fc1-0129-456d-acd6-8479f7c3f338 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:50.536229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-80ee9fc1-0129-456d-acd6-8479f7c3f338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:50.632118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 592/1163] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:45:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:50.935996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc473ac2-b9c4-45e7-8ac7-571e06fbf0b7 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:50.937600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-fc473ac2-b9c4-45e7-8ac7-571e06fbf0b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:51.045229 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc473ac2-b9c4-45e7-8ac7-571e06fbf0b7 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:51.118715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:51.120678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c18227-4e6c-4b2f-aa46-9fb67c442765 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:51.121393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c18227-4e6c-4b2f-aa46-9fb67c442765 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:51.121859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c18227-4e6c-4b2f-aa46-9fb67c442765 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:51.122319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c18227-4e6c-4b2f-aa46-9fb67c442765 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:51.124712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-89c18227-4e6c-4b2f-aa46-9fb67c442765 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:51.233222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c18227-4e6c-4b2f-aa46-9fb67c442765 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:51.234656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c18227-4e6c-4b2f-aa46-9fb67c442765 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:51.253051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 572/1164] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:53.701345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b948167-f022-4f7d-8342-c401b32a51e7 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:53.702021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b948167-f022-4f7d-8342-c401b32a51e7 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:53.702021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b948167-f022-4f7d-8342-c401b32a51e7 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:53.703234 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 593/1165] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:45:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:45:53.708696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8ba8c4-460f-4353-9a6c-18106e3df5d5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:53.709125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8ba8c4-460f-4353-9a6c-18106e3df5d5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:53.709254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8ba8c4-460f-4353-9a6c-18106e3df5d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:53.777223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-ac8ba8c4-460f-4353-9a6c-18106e3df5d5 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:53.778426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ac8ba8c4-460f-4353-9a6c-18106e3df5d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:53.833769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:53.878698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 573/1166] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:45:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:54.177529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b948167-f022-4f7d-8342-c401b32a51e7 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:54.179032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8b948167-f022-4f7d-8342-c401b32a51e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:54.215850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:54.285206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b948167-f022-4f7d-8342-c401b32a51e7 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:54.357927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5954f18-2987-412a-b85c-fd7e55b51acc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:54.358526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5954f18-2987-412a-b85c-fd7e55b51acc service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:54.358808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5954f18-2987-412a-b85c-fd7e55b51acc service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:54.359085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5954f18-2987-412a-b85c-fd7e55b51acc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:54.360689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c5954f18-2987-412a-b85c-fd7e55b51acc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:54.442422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:54.472864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5954f18-2987-412a-b85c-fd7e55b51acc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:54.474282 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5954f18-2987-412a-b85c-fd7e55b51acc service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:54.492780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 594/1167] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:45:56.892447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428ee57-c897-4003-a0b6-8f79319ccf95 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:56.892974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428ee57-c897-4003-a0b6-8f79319ccf95 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:56.893094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428ee57-c897-4003-a0b6-8f79319ccf95 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:56.894377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 574/1168] 10.222.0.39 () {54 vars in 927 bytes} [Mon Oct 24 08:45:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:45:56.899792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479a85b-5fc8-46ba-a4ba-7cbe902acfc6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:56.900270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479a85b-5fc8-46ba-a4ba-7cbe902acfc6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:56.900376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479a85b-5fc8-46ba-a4ba-7cbe902acfc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:56.966850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0479a85b-5fc8-46ba-a4ba-7cbe902acfc6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:45:56.968103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0479a85b-5fc8-46ba-a4ba-7cbe902acfc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:57.071147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 595/1169] 10.222.0.39 () {58 vars in 1024 bytes} [Mon Oct 24 08:45:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:45:57.363773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e428ee57-c897-4003-a0b6-8f79319ccf95 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:45:57.365382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e428ee57-c897-4003-a0b6-8f79319ccf95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:57.469173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e428ee57-c897-4003-a0b6-8f79319ccf95 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:45:57.547411 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98dfa98-2df8-46ab-93c0-1498c11cbdda service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:45:57.548012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98dfa98-2df8-46ab-93c0-1498c11cbdda service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:45:57.548318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98dfa98-2df8-46ab-93c0-1498c11cbdda service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:45:57.548594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98dfa98-2df8-46ab-93c0-1498c11cbdda service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:45:57.550183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a98dfa98-2df8-46ab-93c0-1498c11cbdda service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:45:57.558688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:45:57.660433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98dfa98-2df8-46ab-93c0-1498c11cbdda service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:45:57.661866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98dfa98-2df8-46ab-93c0-1498c11cbdda service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:45:57.680490 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 575/1170] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:45:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:46:07.069625 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=82 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:46:07.678192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=50 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:12.218774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5dfd3-a0b3-4fc5-a8a4-aa2eb5450703 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:12.219365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5dfd3-a0b3-4fc5-a8a4-aa2eb5450703 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:12.219365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5dfd3-a0b3-4fc5-a8a4-aa2eb5450703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:12.297038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4bb5dfd3-a0b3-4fc5-a8a4-aa2eb5450703 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:47:12.298238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4bb5dfd3-a0b3-4fc5-a8a4-aa2eb5450703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:12.395927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 596/1171] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:47:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:47:12.404684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c2e5a6-0161-4b0e-b370-b06ee4f1f465 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:12.405273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c2e5a6-0161-4b0e-b370-b06ee4f1f465 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:12.405273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c2e5a6-0161-4b0e-b370-b06ee4f1f465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:12.490425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a1c2e5a6-0161-4b0e-b370-b06ee4f1f465 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:47:12.491687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a1c2e5a6-0161-4b0e-b370-b06ee4f1f465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:12.589976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 576/1172] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:47:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:47:12.594804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb5dfd3-a0b3-4fc5-a8a4-aa2eb5450703 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:12.596049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4bb5dfd3-a0b3-4fc5-a8a4-aa2eb5450703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:12.607578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb5dfd3-a0b3-4fc5-a8a4-aa2eb5450703 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:12.671671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:12.710608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:12.711209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:12.711483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:12.711749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:12.712422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:47:12.712780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:12.713722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:47:12.713722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:47:12.713722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:47:12.714615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:12.726100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 597/1173] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:47:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:47:12.736184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c2e5a6-0161-4b0e-b370-b06ee4f1f465 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:12.746162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c2e5a6-0161-4b0e-b370-b06ee4f1f465 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:12.809787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:12.844630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:12.845230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:12.845507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:12.845775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:12.846399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:12.847499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING py.warnings [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:47:12.847499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: warnings.warn(msg) Oct 24 08:47:12.847499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: Oct 24 08:47:12.848707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:12.908217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 577/1174] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:47:12 2022] POST /identity/v3/projects => generated 377 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:47:12.912655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:12.914208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:12.949684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ceb86-e202-4383-a8bc-3c136a4fdbfc admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:13.047278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:13.047793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:13.048071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:13.048365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:13.048999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:13.050003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:47:13.050003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:47:13.050003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:47:13.051192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:13.095262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 598/1175] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:47:12 2022] POST /identity/v3/users => generated 289 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:47:13.098663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:13.100194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:13.167616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301acf10-67e5-4afe-83c3-ed71e16f1bdc admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:13.267739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:13.280386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:13.281034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:13.281315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:13.281590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:13.282266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:13.283865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:13.297406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 578/1176] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:47:13 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:47:13.301846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:13.303368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:13.315159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f8789-9dbe-429a-b1e1-2329ae009a3a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:13.342447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:13.415410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbcb471-f32c-409f-8bf5-ed101893855d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:13.416537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcb471-f32c-409f-8bf5-ed101893855d admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:13.416817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcb471-f32c-409f-8bf5-ed101893855d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:13.417088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcb471-f32c-409f-8bf5-ed101893855d admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:13.417723 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbcb471-f32c-409f-8bf5-ed101893855d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:13.419302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbcb471-f32c-409f-8bf5-ed101893855d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:13.431827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 599/1177] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:47:13 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:47:13.436160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:13.437704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:13.471235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74204518-3023-476b-aea3-8496c94a92f0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:13.570669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81803171-3935-48b8-aeaa-4c1cf2cf54f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:13.571241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803171-3935-48b8-aeaa-4c1cf2cf54f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:13.571526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803171-3935-48b8-aeaa-4c1cf2cf54f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:13.572002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803171-3935-48b8-aeaa-4c1cf2cf54f1 admin admin] PATH_INFO: `/v3/projects/48d32c371b0840c0a47aa6b970b9b2cf/users/3465e5914ac448e2b41affac95549a42/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:13.599274 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81803171-3935-48b8-aeaa-4c1cf2cf54f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48d32c371b0840c0a47aa6b970b9b2cf, user_id=3465e5914ac448e2b41affac95549a42, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:13.602008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81803171-3935-48b8-aeaa-4c1cf2cf54f1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:13.628509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 579/1178] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:47:13 2022] PUT /identity/v3/projects/48d32c371b0840c0a47aa6b970b9b2cf/users/3465e5914ac448e2b41affac95549a42/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:47:13.640581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc6954f-a5ed-463c-ba58-28cd809c8139 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:13.640866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc6954f-a5ed-463c-ba58-28cd809c8139 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:13.641138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc6954f-a5ed-463c-ba58-28cd809c8139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:13.703875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0fc6954f-a5ed-463c-ba58-28cd809c8139 None None] MFA Rules not processed for user `3465e5914ac448e2b41affac95549a42`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:47:13.705127 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0fc6954f-a5ed-463c-ba58-28cd809c8139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:13.784837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 600/1179] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:47:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:47:13.798400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a45c1-7a95-4297-ac57-0a88737d5282 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:13.798684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a45c1-7a95-4297-ac57-0a88737d5282 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:13.798955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a45c1-7a95-4297-ac57-0a88737d5282 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:13.800318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 580/1180] 10.222.0.39 () {54 vars in 963 bytes} [Mon Oct 24 08:47:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:47:13.805263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658325ba-3814-4aa3-94fb-bfb057527d88 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:13.805544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658325ba-3814-4aa3-94fb-bfb057527d88 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:13.805812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658325ba-3814-4aa3-94fb-bfb057527d88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:13.839803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:13.889372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-658325ba-3814-4aa3-94fb-bfb057527d88 None None] MFA Rules not processed for user `412f520f90564e44819e8bf947992efb`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:47:13.890579 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-658325ba-3814-4aa3-94fb-bfb057527d88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:13.968060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 601/1181] 10.222.0.39 () {58 vars in 1060 bytes} [Mon Oct 24 08:47:13 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:47:13.977040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7a45c1-7a95-4297-ac57-0a88737d5282 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:13.980791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9b7a45c1-7a95-4297-ac57-0a88737d5282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:14.035807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:14.091792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7a45c1-7a95-4297-ac57-0a88737d5282 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:14.168380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:14.168985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:14.169261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:14.169528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:14.171122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:14.270574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:14.272018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:14.275582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:14.290598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 581/1182] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:47:13 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:47:14.337303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b505a6ed-e682-4ba4-82a3-c4509e9c0133 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:14.337878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b505a6ed-e682-4ba4-82a3-c4509e9c0133 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:14.337878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b505a6ed-e682-4ba4-82a3-c4509e9c0133 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:14.339144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 602/1183] 10.222.0.39 () {52 vars in 830 bytes} [Mon Oct 24 08:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:47:14.342813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:14.343965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:14.443810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d578e517-c0de-48bf-85fd-a4129deca9ed service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:14.543727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:14.544503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:14.544779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:14.545041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] PATH_INFO: `/v3/users/3465e5914ac448e2b41affac95549a42` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:14.558853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3465e5914ac448e2b41affac95549a42)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:14.559883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING py.warnings [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:47:14.559883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: warnings.warn(msg) Oct 24 08:47:14.559883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: Oct 24 08:47:14.561133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:14.570985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:14.631280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3465e5914ac448e2b41affac95549a42'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:14.645721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3465e5914ac448e2b41affac95549a42'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:14.670489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3465e5914ac448e2b41affac95549a42'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:14.671099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3465e5914ac448e2b41affac95549a42'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:14.671520 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3465e5914ac448e2b41affac95549a42'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:14.688064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 582/1184] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:47:14 2022] DELETE /identity/v3/users/3465e5914ac448e2b41affac95549a42 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:47:14.692790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33daadb-8447-4d52-b660-697a6edd5337 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:14.692969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33daadb-8447-4d52-b660-697a6edd5337 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:14.693148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33daadb-8447-4d52-b660-697a6edd5337 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:14.755376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b33daadb-8447-4d52-b660-697a6edd5337 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:47:14.756621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b33daadb-8447-4d52-b660-697a6edd5337 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:14.835741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:14.849721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 603/1185] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:47:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:47:14.860467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:14.861726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:14.967699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a638657b-2dca-446e-9aba-638be588108a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:15.028700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:15.050972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6799bc62-35f1-49bb-a469-9f86b4b83c96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:15.051642 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6799bc62-35f1-49bb-a469-9f86b4b83c96 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:15.051966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6799bc62-35f1-49bb-a469-9f86b4b83c96 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:15.052329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6799bc62-35f1-49bb-a469-9f86b4b83c96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:15.054020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6799bc62-35f1-49bb-a469-9f86b4b83c96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:15.173851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6799bc62-35f1-49bb-a469-9f86b4b83c96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:15.175313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6799bc62-35f1-49bb-a469-9f86b4b83c96 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:15.195017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 583/1186] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:47:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:47:15.562724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33daadb-8447-4d52-b660-697a6edd5337 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:47:15.564053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b33daadb-8447-4d52-b660-697a6edd5337 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:47:15.576253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33daadb-8447-4d52-b660-697a6edd5337 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:47:15.677695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:47:15.678326 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:47:15.678603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:47:15.678866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] PATH_INFO: `/v3/projects/48d32c371b0840c0a47aa6b970b9b2cf` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:47:15.690041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=48d32c371b0840c0a47aa6b970b9b2cf)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:47:15.691059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:47:15.691059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:47:15.691059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:47:15.692307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:47:15.750761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48d32c371b0840c0a47aa6b970b9b2cf', 'request_id': 'req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:15.765118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48d32c371b0840c0a47aa6b970b9b2cf', 'request_id': 'req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:15.775263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48d32c371b0840c0a47aa6b970b9b2cf', 'request_id': 'req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:15.799752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] The token cache is being invalidate because project 48d32c371b0840c0a47aa6b970b9b2cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:47:15.800044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:15.800509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afeb0ebe-b8a6-4839-82c9-98cd1f546ab4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:47:15.801922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 604/1187] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:47:15 2022] DELETE /identity/v3/projects/48d32c371b0840c0a47aa6b970b9b2cf => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:47:25.192650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=62 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:47:25.797684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:48:46.371063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b609fdc7-e282-418a-9bc0-cf4f432efb30 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:46.371656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b609fdc7-e282-418a-9bc0-cf4f432efb30 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:46.371656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b609fdc7-e282-418a-9bc0-cf4f432efb30 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:46.372988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 584/1188] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:48:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:48:46.377915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723eb19-57bf-4da4-9da8-1ec3bd2bd7b3 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:46.378463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723eb19-57bf-4da4-9da8-1ec3bd2bd7b3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:46.378463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723eb19-57bf-4da4-9da8-1ec3bd2bd7b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:46.463845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a723eb19-57bf-4da4-9da8-1ec3bd2bd7b3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:48:46.465133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a723eb19-57bf-4da4-9da8-1ec3bd2bd7b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:48:46.569507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 605/1189] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:48:46.734994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b609fdc7-e282-418a-9bc0-cf4f432efb30 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:48:46.737143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b609fdc7-e282-418a-9bc0-cf4f432efb30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:48:46.859913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b609fdc7-e282-418a-9bc0-cf4f432efb30 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:48:46.866253 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39191136-c077-44f1-8c9a-3c58d14ffb25 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:48:46.866857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39191136-c077-44f1-8c9a-3c58d14ffb25 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:46.867140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39191136-c077-44f1-8c9a-3c58d14ffb25 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:46.867665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39191136-c077-44f1-8c9a-3c58d14ffb25 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:46.875942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39191136-c077-44f1-8c9a-3c58d14ffb25 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:48:46.877573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39191136-c077-44f1-8c9a-3c58d14ffb25 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:48:46.887696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 585/1190] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:48:48.019954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2be4e8-6474-4ac9-b19b-a7a19869d901 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:48.020599 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2be4e8-6474-4ac9-b19b-a7a19869d901 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:48.020599 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2be4e8-6474-4ac9-b19b-a7a19869d901 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:48.022677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 606/1191] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:48:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:48:48.027760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cbec30-f8b7-42ec-8f25-6d90c85cdd82 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:48.028249 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cbec30-f8b7-42ec-8f25-6d90c85cdd82 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:48.028394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cbec30-f8b7-42ec-8f25-6d90c85cdd82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:48.048924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-99cbec30-f8b7-42ec-8f25-6d90c85cdd82 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:48:48.050153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-99cbec30-f8b7-42ec-8f25-6d90c85cdd82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:48:48.057994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 586/1192] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:48:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:48:48.075043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8d19ab-8d32-41af-a900-20f10ea74a1f None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:48.075480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8d19ab-8d32-41af-a900-20f10ea74a1f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:48.075886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8d19ab-8d32-41af-a900-20f10ea74a1f None None] PATH_INFO: `/v3` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:48.078335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 607/1193] 10.222.0.13 () {54 vars in 925 bytes} [Mon Oct 24 08:48:48 2022] GET /identity/v3 => generated 255 bytes in 5 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:48:48.086524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6dfbb6-571d-4bc5-ab6f-a0e6b9c8fe7c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:48.086849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6dfbb6-571d-4bc5-ab6f-a0e6b9c8fe7c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:48.087131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6dfbb6-571d-4bc5-ab6f-a0e6b9c8fe7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:48.158243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5d6dfbb6-571d-4bc5-ab6f-a0e6b9c8fe7c None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:48:48.158243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5d6dfbb6-571d-4bc5-ab6f-a0e6b9c8fe7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:48:48.215696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8d19ab-8d32-41af-a900-20f10ea74a1f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:48:48.220577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:48:48.231009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8d19ab-8d32-41af-a900-20f10ea74a1f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:48:48.240126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617898d0-2a56-4bb9-8ad8-ab7bb3e8d4c1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:48:48.240882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617898d0-2a56-4bb9-8ad8-ab7bb3e8d4c1 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:48.241294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617898d0-2a56-4bb9-8ad8-ab7bb3e8d4c1 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:48.241710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617898d0-2a56-4bb9-8ad8-ab7bb3e8d4c1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:48.243086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 587/1194] 10.222.0.13 () {58 vars in 1015 bytes} [Mon Oct 24 08:48:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:48:48.252557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617898d0-2a56-4bb9-8ad8-ab7bb3e8d4c1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:48:48.254157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617898d0-2a56-4bb9-8ad8-ab7bb3e8d4c1 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:48:48.266509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 608/1195] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:48:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:48:57.263701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c40013-0b04-4c43-8124-ba202d31b9bb None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:57.264304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c40013-0b04-4c43-8124-ba202d31b9bb None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:57.264304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c40013-0b04-4c43-8124-ba202d31b9bb None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:57.265646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 588/1196] 10.222.0.39 () {54 vars in 961 bytes} [Mon Oct 24 08:48:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:48:57.270956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65548a4e-342e-4fe1-9cd2-6edd1fff5fab None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:57.271541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65548a4e-342e-4fe1-9cd2-6edd1fff5fab None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:57.271541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65548a4e-342e-4fe1-9cd2-6edd1fff5fab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:57.292495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-65548a4e-342e-4fe1-9cd2-6edd1fff5fab None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:48:57.293714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-65548a4e-342e-4fe1-9cd2-6edd1fff5fab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:48:57.302782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 609/1197] 10.222.0.39 () {58 vars in 1058 bytes} [Mon Oct 24 08:48:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:48:57.312111 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c40013-0b04-4c43-8124-ba202d31b9bb None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:48:57.326492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c40013-0b04-4c43-8124-ba202d31b9bb None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:48:57.331727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51406cdf-c5dd-4d3f-915f-7ae38c1d73ea service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:48:57.332360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51406cdf-c5dd-4d3f-915f-7ae38c1d73ea service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:57.332641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51406cdf-c5dd-4d3f-915f-7ae38c1d73ea service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:57.332905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51406cdf-c5dd-4d3f-915f-7ae38c1d73ea service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:57.342431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51406cdf-c5dd-4d3f-915f-7ae38c1d73ea service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:48:57.343995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51406cdf-c5dd-4d3f-915f-7ae38c1d73ea service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:48:57.354885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 589/1198] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:48:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:48:59.776954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2b8776-b809-42c7-b624-36a500909d6a None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:59.777587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2b8776-b809-42c7-b624-36a500909d6a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:59.777587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2b8776-b809-42c7-b624-36a500909d6a None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:59.778910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 610/1199] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:48:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:48:59.783850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaeac8c-3e90-4c9b-a808-fd1298613b51 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:48:59.784391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaeac8c-3e90-4c9b-a808-fd1298613b51 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:48:59.784506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaeac8c-3e90-4c9b-a808-fd1298613b51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:48:59.805351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-deaeac8c-3e90-4c9b-a808-fd1298613b51 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:48:59.806668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-deaeac8c-3e90-4c9b-a808-fd1298613b51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:48:59.814654 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 590/1200] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:48:59.978080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2b8776-b809-42c7-b624-36a500909d6a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:48:59.996002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2b8776-b809-42c7-b624-36a500909d6a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:49:00.001527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03f187d-ed2c-47e5-b840-dc6da305dad6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:49:00.002129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f187d-ed2c-47e5-b840-dc6da305dad6 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:00.002409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f187d-ed2c-47e5-b840-dc6da305dad6 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:00.002682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f187d-ed2c-47e5-b840-dc6da305dad6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:00.011576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03f187d-ed2c-47e5-b840-dc6da305dad6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:49:00.013179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03f187d-ed2c-47e5-b840-dc6da305dad6 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:49:00.022450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 611/1201] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:48:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:49:01.114251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2771a3a-0938-48c9-aecc-c33ce30e3ac2 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:01.115029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2771a3a-0938-48c9-aecc-c33ce30e3ac2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:01.115029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2771a3a-0938-48c9-aecc-c33ce30e3ac2 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:01.116185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 591/1202] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 24 08:49:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:49:01.122481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7407e25-ad24-48bb-957d-5e67af945362 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:01.123028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7407e25-ad24-48bb-957d-5e67af945362 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:01.123137 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7407e25-ad24-48bb-957d-5e67af945362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:01.142544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b7407e25-ad24-48bb-957d-5e67af945362 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:49:01.143752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b7407e25-ad24-48bb-957d-5e67af945362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:01.152559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 612/1203] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 24 08:49:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:49:01.307438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2771a3a-0938-48c9-aecc-c33ce30e3ac2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:49:01.321469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2771a3a-0938-48c9-aecc-c33ce30e3ac2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:49:01.326961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03883e58-cb2e-4e52-9d5e-f250528f2f13 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:49:01.327587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03883e58-cb2e-4e52-9d5e-f250528f2f13 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:01.327901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03883e58-cb2e-4e52-9d5e-f250528f2f13 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:01.328202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03883e58-cb2e-4e52-9d5e-f250528f2f13 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:01.337099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03883e58-cb2e-4e52-9d5e-f250528f2f13 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:49:01.338670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03883e58-cb2e-4e52-9d5e-f250528f2f13 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:49:01.347957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 592/1204] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:49:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:49:05.522976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466c24fe-9624-40cf-9ca2-d03715984d0c None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:05.523628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466c24fe-9624-40cf-9ca2-d03715984d0c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:05.523628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466c24fe-9624-40cf-9ca2-d03715984d0c None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:05.524914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 613/1205] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:49:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:49:05.530577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e3e0ab-4063-45e2-86fb-63819b9145c8 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:05.531140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e3e0ab-4063-45e2-86fb-63819b9145c8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:05.531140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e3e0ab-4063-45e2-86fb-63819b9145c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:05.551974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-81e3e0ab-4063-45e2-86fb-63819b9145c8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:49:05.553232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-81e3e0ab-4063-45e2-86fb-63819b9145c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:05.562194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 593/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:49:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:49:05.870129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466c24fe-9624-40cf-9ca2-d03715984d0c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:49:05.872175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-466c24fe-9624-40cf-9ca2-d03715984d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:05.909774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:49:05.962833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466c24fe-9624-40cf-9ca2-d03715984d0c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:49:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d7c2d8-d2fd-427e-a101-f93e4e0b8244 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:49:05.967535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d7c2d8-d2fd-427e-a101-f93e4e0b8244 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:05.967816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d7c2d8-d2fd-427e-a101-f93e4e0b8244 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:05.968094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d7c2d8-d2fd-427e-a101-f93e4e0b8244 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:05.981772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d7c2d8-d2fd-427e-a101-f93e4e0b8244 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:49:05.984086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d7c2d8-d2fd-427e-a101-f93e4e0b8244 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:49:05.997192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 614/1207] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:49:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:49:08.298606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88da5e5-226d-4fb3-ac23-93c29df1ad61 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:08.299168 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88da5e5-226d-4fb3-ac23-93c29df1ad61 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:08.299168 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88da5e5-226d-4fb3-ac23-93c29df1ad61 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:08.300606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 594/1208] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:49:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:49:08.306790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d756b-4ce6-48f3-9305-ed953031b766 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:08.307232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d756b-4ce6-48f3-9305-ed953031b766 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:08.307339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d756b-4ce6-48f3-9305-ed953031b766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:08.327123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-791d756b-4ce6-48f3-9305-ed953031b766 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:49:08.328345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-791d756b-4ce6-48f3-9305-ed953031b766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:08.336563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 615/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:49:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:49:08.626871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88da5e5-226d-4fb3-ac23-93c29df1ad61 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:49:08.628603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a88da5e5-226d-4fb3-ac23-93c29df1ad61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:08.642071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88da5e5-226d-4fb3-ac23-93c29df1ad61 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:49:08.645589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a999ffaa-78aa-4138-876f-c579a5b93188 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:49:08.646172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999ffaa-78aa-4138-876f-c579a5b93188 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:08.646453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999ffaa-78aa-4138-876f-c579a5b93188 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:08.646730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999ffaa-78aa-4138-876f-c579a5b93188 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:08.655574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a999ffaa-78aa-4138-876f-c579a5b93188 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:49:08.657080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a999ffaa-78aa-4138-876f-c579a5b93188 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:49:08.667308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 595/1210] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:49:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:49:12.456429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e72c021-9037-4b04-a578-e7dfa6d2285c None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:12.456971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e72c021-9037-4b04-a578-e7dfa6d2285c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:12.456971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e72c021-9037-4b04-a578-e7dfa6d2285c None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:12.458376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 616/1211] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:49:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:49:12.465217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfcaa57-155a-4c75-8465-836f8bc6d3d4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:12.465757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfcaa57-155a-4c75-8465-836f8bc6d3d4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:12.465757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfcaa57-155a-4c75-8465-836f8bc6d3d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:12.486746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3dfcaa57-155a-4c75-8465-836f8bc6d3d4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:49:12.487954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3dfcaa57-155a-4c75-8465-836f8bc6d3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:12.496556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 596/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:49:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:49:12.815582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e72c021-9037-4b04-a578-e7dfa6d2285c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:49:12.817483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9e72c021-9037-4b04-a578-e7dfa6d2285c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:12.914661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e72c021-9037-4b04-a578-e7dfa6d2285c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:49:12.920395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba012e0-45a4-4716-a947-2dfffdaa594d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:49:12.921435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba012e0-45a4-4716-a947-2dfffdaa594d service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:12.921939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba012e0-45a4-4716-a947-2dfffdaa594d service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:12.922425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba012e0-45a4-4716-a947-2dfffdaa594d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:12.938771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba012e0-45a4-4716-a947-2dfffdaa594d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:49:12.940770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba012e0-45a4-4716-a947-2dfffdaa594d service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:49:12.955689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 617/1213] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:49:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:49:15.211086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c80356-f81d-4d1d-bafe-61369c2c951f None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:15.211086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c80356-f81d-4d1d-bafe-61369c2c951f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:15.211758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c80356-f81d-4d1d-bafe-61369c2c951f None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:15.212761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 597/1214] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 24 08:49:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:49:15.218149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca7501-6afb-4220-817d-96ef8098418c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:15.218583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca7501-6afb-4220-817d-96ef8098418c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:15.218690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca7501-6afb-4220-817d-96ef8098418c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:15.238535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fbca7501-6afb-4220-817d-96ef8098418c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:49:15.239758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fbca7501-6afb-4220-817d-96ef8098418c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:15.250371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 618/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 24 08:49:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:49:15.550612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c80356-f81d-4d1d-bafe-61369c2c951f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:49:15.552305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f2c80356-f81d-4d1d-bafe-61369c2c951f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:49:15.566060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c80356-f81d-4d1d-bafe-61369c2c951f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:49:15.570029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7345f823-2b60-44b3-8bee-6f804272653a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:49:15.570615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345f823-2b60-44b3-8bee-6f804272653a service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:49:15.570892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345f823-2b60-44b3-8bee-6f804272653a service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:49:15.571163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345f823-2b60-44b3-8bee-6f804272653a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:49:15.579872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7345f823-2b60-44b3-8bee-6f804272653a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:49:15.581319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7345f823-2b60-44b3-8bee-6f804272653a service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:49:15.591778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 598/1216] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:49:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:49:25.233895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=63 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:49:25.587438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=37 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:41.164212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833d30ae-ed5b-4d3c-8fe1-a5695a64657c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.164902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833d30ae-ed5b-4d3c-8fe1-a5695a64657c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.164902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833d30ae-ed5b-4d3c-8fe1-a5695a64657c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.196487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-833d30ae-ed5b-4d3c-8fe1-a5695a64657c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:41.197710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-833d30ae-ed5b-4d3c-8fe1-a5695a64657c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:41.213626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 619/1217] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:41.226022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f93e1a6-65c6-4042-929b-b750d14d2ad1 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.227184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f93e1a6-65c6-4042-929b-b750d14d2ad1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.227687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f93e1a6-65c6-4042-929b-b750d14d2ad1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.259495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7f93e1a6-65c6-4042-929b-b750d14d2ad1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:41.261115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7f93e1a6-65c6-4042-929b-b750d14d2ad1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:41.276191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 599/1218] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:41.281097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833d30ae-ed5b-4d3c-8fe1-a5695a64657c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:41.297630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833d30ae-ed5b-4d3c-8fe1-a5695a64657c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:41.308318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69150827-cd3f-484c-b90f-1dbaeda3950d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:41.309147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69150827-cd3f-484c-b90f-1dbaeda3950d admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.309583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69150827-cd3f-484c-b90f-1dbaeda3950d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.309926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69150827-cd3f-484c-b90f-1dbaeda3950d admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.310619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69150827-cd3f-484c-b90f-1dbaeda3950d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:41.311039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69150827-cd3f-484c-b90f-1dbaeda3950d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:41.313165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69150827-cd3f-484c-b90f-1dbaeda3950d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:41.338089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 620/1219] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:50:41.351717 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f93e1a6-65c6-4042-929b-b750d14d2ad1 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:41.366521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f93e1a6-65c6-4042-929b-b750d14d2ad1 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:41.377722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9afba-32bc-4957-bf10-01b92a5a3cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:41.378742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9afba-32bc-4957-bf10-01b92a5a3cdc admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.378889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7072aa-7792-4ced-8adb-9b11a16821fe None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.379138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7072aa-7792-4ced-8adb-9b11a16821fe None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.379418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7072aa-7792-4ced-8adb-9b11a16821fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.379946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9afba-32bc-4957-bf10-01b92a5a3cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.380411 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9afba-32bc-4957-bf10-01b92a5a3cdc admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.381658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9afba-32bc-4957-bf10-01b92a5a3cdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:41.384072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9afba-32bc-4957-bf10-01b92a5a3cdc admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:41.410245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fe7072aa-7792-4ced-8adb-9b11a16821fe None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:41.411572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fe7072aa-7792-4ced-8adb-9b11a16821fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:41.428187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 621/1220] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:50:41.438765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20350a3c-a28b-47d3-80c1-9a5e0379f00d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.439973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20350a3c-a28b-47d3-80c1-9a5e0379f00d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.440442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20350a3c-a28b-47d3-80c1-9a5e0379f00d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.450460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 600/1221] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/projects => generated 405 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:41.454759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9afba-32bc-4957-bf10-01b92a5a3cdc admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:41.477830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-20350a3c-a28b-47d3-80c1-9a5e0379f00d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:41.479437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-20350a3c-a28b-47d3-80c1-9a5e0379f00d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:41.578817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9afba-32bc-4957-bf10-01b92a5a3cdc admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:41.585960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 622/1222] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:41.591830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20350a3c-a28b-47d3-80c1-9a5e0379f00d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:41.598510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:41.599731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.599731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.600254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.601202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:41.602652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING py.warnings [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:50:41.602652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: warnings.warn(msg) Oct 24 08:50:41.602652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: Oct 24 08:50:41.604121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:41.608861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20350a3c-a28b-47d3-80c1-9a5e0379f00d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:41.624984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:41.625937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.626374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.627326 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.628171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:41.628724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:41.630626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:41.648148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 623/1223] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:50:41.660938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:41.663102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 601/1224] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/users => generated 303 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:50:41.667577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:41.679729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2579899-bf86-4e89-816d-9df1cd1285ce admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:41.685410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:41.686278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.686729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.687162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.687973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:41.688432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a49b9d-0500-41f6-bd61-2fe8ac2ba298 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:41.690035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:50:41.690035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:50:41.690035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:50:41.691757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:41.714745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211639a9-0b08-4be6-9203-578a9f8610fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:41.715509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211639a9-0b08-4be6-9203-578a9f8610fa admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.716147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211639a9-0b08-4be6-9203-578a9f8610fa admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.716537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211639a9-0b08-4be6-9203-578a9f8610fa admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.717279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211639a9-0b08-4be6-9203-578a9f8610fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:41.718998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211639a9-0b08-4be6-9203-578a9f8610fa admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:41.738260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 602/1225] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:41 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:50:41.739715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab4f40-3edf-4cf1-9479-3fd48133c687 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.740098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab4f40-3edf-4cf1-9479-3fd48133c687 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.741619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 624/1226] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/projects => generated 371 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:50:41.744712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:41.745553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab4f40-3edf-4cf1-9479-3fd48133c687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.795693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-84ab4f40-3edf-4cf1-9479-3fd48133c687 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:41.797595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-84ab4f40-3edf-4cf1-9479-3fd48133c687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:41.857185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aa607d-c9f0-4aba-bbf5-880fb28a315f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:41.868451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18828555-e328-4237-9d51-880dd5abc482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:41.869372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18828555-e328-4237-9d51-880dd5abc482 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.869831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18828555-e328-4237-9d51-880dd5abc482 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.870468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18828555-e328-4237-9d51-880dd5abc482 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.871308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18828555-e328-4237-9d51-880dd5abc482 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:41.878507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18828555-e328-4237-9d51-880dd5abc482 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:41.894691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 625/1227] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:41 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:50:41.896492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18828555-e328-4237-9d51-880dd5abc482 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:41.916793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18828555-e328-4237-9d51-880dd5abc482 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:41.935991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdbf398-f013-4245-971c-b3bcdaa0f448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:41.936943 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdbf398-f013-4245-971c-b3bcdaa0f448 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.937396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdbf398-f013-4245-971c-b3bcdaa0f448 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.937834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdbf398-f013-4245-971c-b3bcdaa0f448 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.938747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdbf398-f013-4245-971c-b3bcdaa0f448 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:41.941165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdbf398-f013-4245-971c-b3bcdaa0f448 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:41.954133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 603/1228] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:41.958244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45a21f6-584f-4543-86e0-43d450b10d9f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:41.958244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45a21f6-584f-4543-86e0-43d450b10d9f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:41.958244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45a21f6-584f-4543-86e0-43d450b10d9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:41.989863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 626/1229] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/users => generated 286 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:41.991003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdbf398-f013-4245-971c-b3bcdaa0f448 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.004543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdbf398-f013-4245-971c-b3bcdaa0f448 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.007726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a45a21f6-584f-4543-86e0-43d450b10d9f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.011812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a45a21f6-584f-4543-86e0-43d450b10d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.012624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3f2a4d-7864-40ff-9119-a400393b6452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.014368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3f2a4d-7864-40ff-9119-a400393b6452 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.015745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3f2a4d-7864-40ff-9119-a400393b6452 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.016209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3f2a4d-7864-40ff-9119-a400393b6452 admin admin] PATH_INFO: `/v3/projects/246835f30eb64b17ac0adccb40379b7f/users/f39e25b9275e431f8a7304171a0c1ffb/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.026663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 604/1230] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.033764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd236764-06db-4c9c-966c-38f89baafdc9 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.034352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd236764-06db-4c9c-966c-38f89baafdc9 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.034907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd236764-06db-4c9c-966c-38f89baafdc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.051580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3f2a4d-7864-40ff-9119-a400393b6452 admin admin] RBAC: Authorizing `identity:create_grant(project_id=246835f30eb64b17ac0adccb40379b7f, user_id=f39e25b9275e431f8a7304171a0c1ffb, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.054592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3f2a4d-7864-40ff-9119-a400393b6452 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.063141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:42.072576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-cd236764-06db-4c9c-966c-38f89baafdc9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.075831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cd236764-06db-4c9c-966c-38f89baafdc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.086293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 627/1231] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:41 2022] PUT /identity/v3/projects/246835f30eb64b17ac0adccb40379b7f/users/f39e25b9275e431f8a7304171a0c1ffb/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:42.087233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3f2a4d-7864-40ff-9119-a400393b6452 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.098719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 605/1232] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.100072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893fc81-4138-4d40-a81c-b0458122b3ae None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.101498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893fc81-4138-4d40-a81c-b0458122b3ae None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.102153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893fc81-4138-4d40-a81c-b0458122b3ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.145788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5893fc81-4138-4d40-a81c-b0458122b3ae None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.147161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5893fc81-4138-4d40-a81c-b0458122b3ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.178930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:42.192858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3f2a4d-7864-40ff-9119-a400393b6452 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.197680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b51940-a250-48f4-b0a9-72cd3fddabce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.198290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b51940-a250-48f4-b0a9-72cd3fddabce admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.198574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b51940-a250-48f4-b0a9-72cd3fddabce admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.198850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b51940-a250-48f4-b0a9-72cd3fddabce admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.199492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b51940-a250-48f4-b0a9-72cd3fddabce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.201117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b51940-a250-48f4-b0a9-72cd3fddabce admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.211601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 628/1233] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:42 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:42.212828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4dc32c-433f-4164-b80c-61f7c025decf None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.213112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4dc32c-433f-4164-b80c-61f7c025decf None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.213388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4dc32c-433f-4164-b80c-61f7c025decf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.232179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 606/1234] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.233452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27db5855-26c6-478d-9182-76325dd68b99 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.233805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27db5855-26c6-478d-9182-76325dd68b99 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.234130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27db5855-26c6-478d-9182-76325dd68b99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.234953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4a4dc32c-433f-4164-b80c-61f7c025decf None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.240311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4a4dc32c-433f-4164-b80c-61f7c025decf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.251490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 629/1235] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.251490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4dc32c-433f-4164-b80c-61f7c025decf None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.258669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4dc32c-433f-4164-b80c-61f7c025decf None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.263569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0016ab-47ff-450c-8faa-642bdbb84df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.264203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0016ab-47ff-450c-8faa-642bdbb84df7 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.264483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0016ab-47ff-450c-8faa-642bdbb84df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.264756 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0016ab-47ff-450c-8faa-642bdbb84df7 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.265404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0016ab-47ff-450c-8faa-642bdbb84df7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:42.265708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0016ab-47ff-450c-8faa-642bdbb84df7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.267297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0016ab-47ff-450c-8faa-642bdbb84df7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.270214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-27db5855-26c6-478d-9182-76325dd68b99 None None] MFA Rules not processed for user `f39e25b9275e431f8a7304171a0c1ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.271511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-27db5855-26c6-478d-9182-76325dd68b99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.280280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 630/1236] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:50:42.284253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e40ca-ac70-4e6c-8ea3-14d04218d357 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.285283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e40ca-ac70-4e6c-8ea3-14d04218d357 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.285736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e40ca-ac70-4e6c-8ea3-14d04218d357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.310162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-1d9e40ca-ac70-4e6c-8ea3-14d04218d357 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.312163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1d9e40ca-ac70-4e6c-8ea3-14d04218d357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.321491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 631/1237] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.323197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad4176-b11a-41f2-a3c5-5ea33178c50f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.323639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad4176-b11a-41f2-a3c5-5ea33178c50f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.324063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad4176-b11a-41f2-a3c5-5ea33178c50f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.350703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d3ad4176-b11a-41f2-a3c5-5ea33178c50f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.352014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d3ad4176-b11a-41f2-a3c5-5ea33178c50f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.357094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 607/1238] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.357942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27db5855-26c6-478d-9182-76325dd68b99 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.360258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 632/1239] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.362369 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe0c596-24bf-42c2-bdac-2faa1ab18a7f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.362720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe0c596-24bf-42c2-bdac-2faa1ab18a7f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.363071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe0c596-24bf-42c2-bdac-2faa1ab18a7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.370648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27db5855-26c6-478d-9182-76325dd68b99 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.375328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28696b13-845b-4b35-92c2-66c6f51abb4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.375960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28696b13-845b-4b35-92c2-66c6f51abb4f admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.376267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28696b13-845b-4b35-92c2-66c6f51abb4f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.376602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28696b13-845b-4b35-92c2-66c6f51abb4f admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.377963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28696b13-845b-4b35-92c2-66c6f51abb4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.379653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28696b13-845b-4b35-92c2-66c6f51abb4f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.391280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-cbe0c596-24bf-42c2-bdac-2faa1ab18a7f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.392176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 608/1240] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:42.393592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28696b13-845b-4b35-92c2-66c6f51abb4f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.394057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-cbe0c596-24bf-42c2-bdac-2faa1ab18a7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.406096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 633/1241] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.406096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe0c596-24bf-42c2-bdac-2faa1ab18a7f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.406351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28696b13-845b-4b35-92c2-66c6f51abb4f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.409397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47865e2e-ad69-4585-b946-fc289f993da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.409985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47865e2e-ad69-4585-b946-fc289f993da0 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.410269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47865e2e-ad69-4585-b946-fc289f993da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.410547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47865e2e-ad69-4585-b946-fc289f993da0 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.411205 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47865e2e-ad69-4585-b946-fc289f993da0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:42.411521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47865e2e-ad69-4585-b946-fc289f993da0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.412672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING py.warnings [None req-47865e2e-ad69-4585-b946-fc289f993da0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:50:42.412672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: warnings.warn(msg) Oct 24 08:50:42.412672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: Oct 24 08:50:42.413619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47865e2e-ad69-4585-b946-fc289f993da0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.414781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe0c596-24bf-42c2-bdac-2faa1ab18a7f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.419413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ebafd2-bba4-4713-b3d1-7bebe8006a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.420008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ebafd2-bba4-4713-b3d1-7bebe8006a88 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.420310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ebafd2-bba4-4713-b3d1-7bebe8006a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.420578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ebafd2-bba4-4713-b3d1-7bebe8006a88 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.421210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ebafd2-bba4-4713-b3d1-7bebe8006a88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.423070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ebafd2-bba4-4713-b3d1-7bebe8006a88 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.430264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 609/1242] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:50:42.431546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bf3d6-a024-4da9-8e18-335e33268704 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.431831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bf3d6-a024-4da9-8e18-335e33268704 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.432102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bf3d6-a024-4da9-8e18-335e33268704 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.453725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7f2bf3d6-a024-4da9-8e18-335e33268704 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.454708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 634/1243] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/projects => generated 377 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:42.455036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7f2bf3d6-a024-4da9-8e18-335e33268704 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.457440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8439f-e68e-4296-9426-ca4e17ef67c2 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.457753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8439f-e68e-4296-9426-ca4e17ef67c2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.458023 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8439f-e68e-4296-9426-ca4e17ef67c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.478303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-81c8439f-e68e-4296-9426-ca4e17ef67c2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.479533 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-81c8439f-e68e-4296-9426-ca4e17ef67c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.523911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 610/1244] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.524855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2bf3d6-a024-4da9-8e18-335e33268704 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.538981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2bf3d6-a024-4da9-8e18-335e33268704 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.543587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d1cd9-5109-42c4-93ea-887be6fbf63a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.544292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d1cd9-5109-42c4-93ea-887be6fbf63a admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.544582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d1cd9-5109-42c4-93ea-887be6fbf63a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.544857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d1cd9-5109-42c4-93ea-887be6fbf63a admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.545601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 635/1245] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.545730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6d1cd9-5109-42c4-93ea-887be6fbf63a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.546944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ab8028-cd1b-4ab3-aa85-df05bfc3d64a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.547227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ab8028-cd1b-4ab3-aa85-df05bfc3d64a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.547444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6d1cd9-5109-42c4-93ea-887be6fbf63a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.547544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ab8028-cd1b-4ab3-aa85-df05bfc3d64a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.568692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b7ab8028-cd1b-4ab3-aa85-df05bfc3d64a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.570253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b7ab8028-cd1b-4ab3-aa85-df05bfc3d64a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.576187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 611/1246] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/projects => generated 403 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:42.576991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d1cd9-5109-42c4-93ea-887be6fbf63a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.578729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 636/1247] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.579684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ab8028-cd1b-4ab3-aa85-df05bfc3d64a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.590984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ab8028-cd1b-4ab3-aa85-df05bfc3d64a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.596103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:42.613197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:42.649875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d1cd9-5109-42c4-93ea-887be6fbf63a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.653486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.654105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.654413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.654529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c304ce-c3e8-44b3-b7cd-1cd5569b509c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.654670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.655157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c304ce-c3e8-44b3-b7cd-1cd5569b509c admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.655312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:42.655473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c304ce-c3e8-44b3-b7cd-1cd5569b509c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.655644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.655752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c304ce-c3e8-44b3-b7cd-1cd5569b509c admin admin] PATH_INFO: `/v3/projects/2488ac7bcf98445a8a25f81c38e18afe/users/03e43b45b5d74ff79f28395b986719f0/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.657304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.669954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 637/1248] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:50:42.670803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.671044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c304ce-c3e8-44b3-b7cd-1cd5569b509c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2488ac7bcf98445a8a25f81c38e18afe, user_id=03e43b45b5d74ff79f28395b986719f0, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.674371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c304ce-c3e8-44b3-b7cd-1cd5569b509c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.681978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a392704-52ed-46f5-bd73-ea2a6cde61f6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.687260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3866bac-d7e2-43bc-9e31-84ede432b22c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.687931 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3866bac-d7e2-43bc-9e31-84ede432b22c admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.688197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3866bac-d7e2-43bc-9e31-84ede432b22c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.688473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3866bac-d7e2-43bc-9e31-84ede432b22c admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.689104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3866bac-d7e2-43bc-9e31-84ede432b22c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.690724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 612/1249] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:42 2022] PUT /identity/v3/projects/2488ac7bcf98445a8a25f81c38e18afe/users/03e43b45b5d74ff79f28395b986719f0/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:42.691585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c304ce-c3e8-44b3-b7cd-1cd5569b509c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.691710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3866bac-d7e2-43bc-9e31-84ede432b22c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.723247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 638/1250] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/projects => generated 391 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:42.724639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669f735-90b9-4eb9-97bb-7708743119d2 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.724928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669f735-90b9-4eb9-97bb-7708743119d2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.725200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669f735-90b9-4eb9-97bb-7708743119d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.746523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4669f735-90b9-4eb9-97bb-7708743119d2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.747730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4669f735-90b9-4eb9-97bb-7708743119d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.766126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c304ce-c3e8-44b3-b7cd-1cd5569b509c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.810404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 639/1251] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:42.811045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4669f735-90b9-4eb9-97bb-7708743119d2 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.821233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4669f735-90b9-4eb9-97bb-7708743119d2 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.825734 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.826388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.826645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.826914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.827543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:42.827841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.829470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.830853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe54747d-535f-4c0e-a749-71b7f5648117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.831474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe54747d-535f-4c0e-a749-71b7f5648117 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.831756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe54747d-535f-4c0e-a749-71b7f5648117 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.832033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe54747d-535f-4c0e-a749-71b7f5648117 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.832752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe54747d-535f-4c0e-a749-71b7f5648117 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.834701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe54747d-535f-4c0e-a749-71b7f5648117 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.841703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 640/1252] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:50:42.842496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.853679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2562d90-00b3-4037-85cc-13530648164f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.859250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49202fd8-db8e-4c3d-8d90-84c451bebb9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.859925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49202fd8-db8e-4c3d-8d90-84c451bebb9f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.860231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49202fd8-db8e-4c3d-8d90-84c451bebb9f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.860507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49202fd8-db8e-4c3d-8d90-84c451bebb9f admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.861645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49202fd8-db8e-4c3d-8d90-84c451bebb9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.864268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49202fd8-db8e-4c3d-8d90-84c451bebb9f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.868469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 613/1253] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/users => generated 289 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:42.872595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe54747d-535f-4c0e-a749-71b7f5648117 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.885023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe54747d-535f-4c0e-a749-71b7f5648117 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.888451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 641/1254] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/users => generated 308 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:42.889240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49202fd8-db8e-4c3d-8d90-84c451bebb9f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.891999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22262068-87b6-491c-ba9c-8de9132a79c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.893179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22262068-87b6-491c-ba9c-8de9132a79c5 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.893179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22262068-87b6-491c-ba9c-8de9132a79c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.893471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22262068-87b6-491c-ba9c-8de9132a79c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.894167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22262068-87b6-491c-ba9c-8de9132a79c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:42.894499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22262068-87b6-491c-ba9c-8de9132a79c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.896104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22262068-87b6-491c-ba9c-8de9132a79c5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.901080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49202fd8-db8e-4c3d-8d90-84c451bebb9f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:42.906032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d737ac-21c5-462b-86bf-71b79d593d15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:42.906659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d737ac-21c5-462b-86bf-71b79d593d15 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.906940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d737ac-21c5-462b-86bf-71b79d593d15 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.907217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d737ac-21c5-462b-86bf-71b79d593d15 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.907916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d737ac-21c5-462b-86bf-71b79d593d15 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:42.909892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 614/1255] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:50:42.911146 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ccf99-ae20-4c1b-82f3-790b39a3dc68 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:42.911430 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ccf99-ae20-4c1b-82f3-790b39a3dc68 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:42.911701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ccf99-ae20-4c1b-82f3-790b39a3dc68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:42.912295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d737ac-21c5-462b-86bf-71b79d593d15 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:42.941674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 642/1256] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/projects => generated 355 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:42.941917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-4a2ccf99-ae20-4c1b-82f3-790b39a3dc68 None None] MFA Rules not processed for user `03e43b45b5d74ff79f28395b986719f0`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:42.942919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d737ac-21c5-462b-86bf-71b79d593d15 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:42.943054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4a2ccf99-ae20-4c1b-82f3-790b39a3dc68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:42.952375 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:43.006984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:43.022332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 615/1257] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:43.023180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2ccf99-ae20-4c1b-82f3-790b39a3dc68 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.026557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d737ac-21c5-462b-86bf-71b79d593d15 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.031273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b4056-ce02-48cc-b465-a468359d63bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.031885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b4056-ce02-48cc-b465-a468359d63bb admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.032313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b4056-ce02-48cc-b465-a468359d63bb admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.032487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b4056-ce02-48cc-b465-a468359d63bb admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.033787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b4056-ce02-48cc-b465-a468359d63bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.034830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2ccf99-ae20-4c1b-82f3-790b39a3dc68 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.035713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b4056-ce02-48cc-b465-a468359d63bb admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.040031 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.040345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.040613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.041244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:43.041540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.043263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.057949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 616/1258] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:50:43.058767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.061183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 643/1259] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:42 2022] POST /identity/v3/users => generated 296 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.062014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b4056-ce02-48cc-b465-a468359d63bb admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.075591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ddad1e-b0ca-4e70-ad0e-2be1fbfab71d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.081569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9fe2df-d445-47d9-9490-1e310eb5736c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.082181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fe2df-d445-47d9-9490-1e310eb5736c admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.082453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fe2df-d445-47d9-9490-1e310eb5736c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.082723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fe2df-d445-47d9-9490-1e310eb5736c admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.083358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9fe2df-d445-47d9-9490-1e310eb5736c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.085292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9fe2df-d445-47d9-9490-1e310eb5736c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.085544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b4056-ce02-48cc-b465-a468359d63bb admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.091470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876e26c1-0bd1-41ca-9998-68db26634913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.092176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876e26c1-0bd1-41ca-9998-68db26634913 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.092550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876e26c1-0bd1-41ca-9998-68db26634913 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.092925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876e26c1-0bd1-41ca-9998-68db26634913 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.093677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876e26c1-0bd1-41ca-9998-68db26634913 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.096529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876e26c1-0bd1-41ca-9998-68db26634913 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.111183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 644/1260] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.113341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876e26c1-0bd1-41ca-9998-68db26634913 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.118535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 617/1261] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/projects => generated 365 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.119418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9fe2df-d445-47d9-9490-1e310eb5736c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876e26c1-0bd1-41ca-9998-68db26634913 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.185635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a2fc66-901f-4ea3-979c-25540a039293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.186417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2fc66-901f-4ea3-979c-25540a039293 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.186417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2fc66-901f-4ea3-979c-25540a039293 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.186652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2fc66-901f-4ea3-979c-25540a039293 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.187411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a2fc66-901f-4ea3-979c-25540a039293 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.189069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a2fc66-901f-4ea3-979c-25540a039293 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.192007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9fe2df-d445-47d9-9490-1e310eb5736c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.196663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b85315-6ade-4a54-9c90-88b397fba334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.197300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b85315-6ade-4a54-9c90-88b397fba334 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.197584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b85315-6ade-4a54-9c90-88b397fba334 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.197858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b85315-6ade-4a54-9c90-88b397fba334 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.198486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b85315-6ade-4a54-9c90-88b397fba334 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.199761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 645/1262] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.200481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b85315-6ade-4a54-9c90-88b397fba334 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.200594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a2fc66-901f-4ea3-979c-25540a039293 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.210871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a2fc66-901f-4ea3-979c-25540a039293 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.216949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74621b2a-eae9-4c10-ab92-898ae1d7be49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.217812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74621b2a-eae9-4c10-ab92-898ae1d7be49 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.218206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74621b2a-eae9-4c10-ab92-898ae1d7be49 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.218471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74621b2a-eae9-4c10-ab92-898ae1d7be49 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.220225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74621b2a-eae9-4c10-ab92-898ae1d7be49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.223912 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74621b2a-eae9-4c10-ab92-898ae1d7be49 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.233905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 618/1263] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/projects => generated 381 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.233905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b85315-6ade-4a54-9c90-88b397fba334 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.248906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 646/1264] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/users => generated 278 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.249640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74621b2a-eae9-4c10-ab92-898ae1d7be49 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.261868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74621b2a-eae9-4c10-ab92-898ae1d7be49 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.312636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b85315-6ade-4a54-9c90-88b397fba334 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.318280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57bbb85-4b72-40d5-b3ff-98989eac3089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.318909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57bbb85-4b72-40d5-b3ff-98989eac3089 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.319188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57bbb85-4b72-40d5-b3ff-98989eac3089 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.319460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57bbb85-4b72-40d5-b3ff-98989eac3089 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.320088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57bbb85-4b72-40d5-b3ff-98989eac3089 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.322015 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57bbb85-4b72-40d5-b3ff-98989eac3089 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.323581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370071e5-430b-445a-a07f-f3571f91adff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.324223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370071e5-430b-445a-a07f-f3571f91adff admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.324510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370071e5-430b-445a-a07f-f3571f91adff admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.324787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370071e5-430b-445a-a07f-f3571f91adff admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.325460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370071e5-430b-445a-a07f-f3571f91adff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.326723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:43.327046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370071e5-430b-445a-a07f-f3571f91adff admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.338734 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 647/1265] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.339784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370071e5-430b-445a-a07f-f3571f91adff admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.352739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370071e5-430b-445a-a07f-f3571f91adff admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.357436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 619/1266] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/projects => generated 369 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.357436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57bbb85-4b72-40d5-b3ff-98989eac3089 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.389596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:43.420636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2753105-a234-409c-95e0-d012347002d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.421556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2753105-a234-409c-95e0-d012347002d2 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.422057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2753105-a234-409c-95e0-d012347002d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.422585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2753105-a234-409c-95e0-d012347002d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.423547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57bbb85-4b72-40d5-b3ff-98989eac3089 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.423739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2753105-a234-409c-95e0-d012347002d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.427523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2753105-a234-409c-95e0-d012347002d2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.428505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2285ad7-fa85-4d3f-a6fb-c12f36473fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.429115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2285ad7-fa85-4d3f-a6fb-c12f36473fdc admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.429400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2285ad7-fa85-4d3f-a6fb-c12f36473fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.429676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2285ad7-fa85-4d3f-a6fb-c12f36473fdc admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.430345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2285ad7-fa85-4d3f-a6fb-c12f36473fdc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.431941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2285ad7-fa85-4d3f-a6fb-c12f36473fdc admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.446321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 620/1267] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.446321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2285ad7-fa85-4d3f-a6fb-c12f36473fdc admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.453995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2285ad7-fa85-4d3f-a6fb-c12f36473fdc admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.460210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e964e2-9e6f-4d2a-9121-a96dfb37012c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.460842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e964e2-9e6f-4d2a-9121-a96dfb37012c admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.461130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e964e2-9e6f-4d2a-9121-a96dfb37012c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.461404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e964e2-9e6f-4d2a-9121-a96dfb37012c admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.462043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e964e2-9e6f-4d2a-9121-a96dfb37012c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.463857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 648/1268] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/projects => generated 387 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.464001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e964e2-9e6f-4d2a-9121-a96dfb37012c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.464708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2753105-a234-409c-95e0-d012347002d2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.487380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 621/1269] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/users => generated 283 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.488222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e964e2-9e6f-4d2a-9121-a96dfb37012c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.498543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e964e2-9e6f-4d2a-9121-a96dfb37012c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.535549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2753105-a234-409c-95e0-d012347002d2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.540075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7706c737-a3ce-4bc1-b729-a771b909afe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.540761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7706c737-a3ce-4bc1-b729-a771b909afe3 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.541038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7706c737-a3ce-4bc1-b729-a771b909afe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.541307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7706c737-a3ce-4bc1-b729-a771b909afe3 admin admin] PATH_INFO: `/v3/projects/2b6daac74a0c422794e65cfdf3bef30d/users/e41f9ffdfc3d4771a917fb25b5417a11/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.558364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97c8a3f-6383-4042-a9b4-460e39f9a705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.558978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97c8a3f-6383-4042-a9b4-460e39f9a705 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.559256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97c8a3f-6383-4042-a9b4-460e39f9a705 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.559491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7706c737-a3ce-4bc1-b729-a771b909afe3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b6daac74a0c422794e65cfdf3bef30d, user_id=e41f9ffdfc3d4771a917fb25b5417a11, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.559596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97c8a3f-6383-4042-a9b4-460e39f9a705 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.560172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97c8a3f-6383-4042-a9b4-460e39f9a705 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.562079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97c8a3f-6383-4042-a9b4-460e39f9a705 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.562205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7706c737-a3ce-4bc1-b729-a771b909afe3 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.576329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 649/1270] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:43 2022] PUT /identity/v3/projects/2b6daac74a0c422794e65cfdf3bef30d/users/e41f9ffdfc3d4771a917fb25b5417a11/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:43.579493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7706c737-a3ce-4bc1-b729-a771b909afe3 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.587346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 622/1271] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/users => generated 291 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.587346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97c8a3f-6383-4042-a9b4-460e39f9a705 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.596541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97c8a3f-6383-4042-a9b4-460e39f9a705 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.646833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:43.648183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7706c737-a3ce-4bc1-b729-a771b909afe3 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.653770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65adbf00-789a-4a66-a0ae-138ddec19ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.654395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65adbf00-789a-4a66-a0ae-138ddec19ab3 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.654675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65adbf00-789a-4a66-a0ae-138ddec19ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.654945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65adbf00-789a-4a66-a0ae-138ddec19ab3 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.655594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65adbf00-789a-4a66-a0ae-138ddec19ab3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.657233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65adbf00-789a-4a66-a0ae-138ddec19ab3 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.660065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e693dca1-d967-4b01-88c9-154f4d50ef35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.660715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e693dca1-d967-4b01-88c9-154f4d50ef35 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.661023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e693dca1-d967-4b01-88c9-154f4d50ef35 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.661298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e693dca1-d967-4b01-88c9-154f4d50ef35 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.661898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e693dca1-d967-4b01-88c9-154f4d50ef35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.663527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e693dca1-d967-4b01-88c9-154f4d50ef35 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.668429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 650/1272] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.669382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65adbf00-789a-4a66-a0ae-138ddec19ab3 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.675626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 623/1273] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.676458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e693dca1-d967-4b01-88c9-154f4d50ef35 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.680412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65adbf00-789a-4a66-a0ae-138ddec19ab3 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.685003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c57d51-a3c0-46bd-b8ab-5bdb6a19d75f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.685593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c57d51-a3c0-46bd-b8ab-5bdb6a19d75f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.685874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c57d51-a3c0-46bd-b8ab-5bdb6a19d75f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.686004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e693dca1-d967-4b01-88c9-154f4d50ef35 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.686146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c57d51-a3c0-46bd-b8ab-5bdb6a19d75f admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.686773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c57d51-a3c0-46bd-b8ab-5bdb6a19d75f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.688703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c57d51-a3c0-46bd-b8ab-5bdb6a19d75f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.691697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4986615b-0cf1-4cd7-88ae-8a912dd4f4b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.692364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4986615b-0cf1-4cd7-88ae-8a912dd4f4b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.692641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4986615b-0cf1-4cd7-88ae-8a912dd4f4b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.692936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4986615b-0cf1-4cd7-88ae-8a912dd4f4b4 admin admin] PATH_INFO: `/v3/projects/25f1ebbfb9d042fab037dbe07a651913/users/701e4507214f4634ad09e32e21749a35/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.711233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4986615b-0cf1-4cd7-88ae-8a912dd4f4b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=25f1ebbfb9d042fab037dbe07a651913, user_id=701e4507214f4634ad09e32e21749a35, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.712494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 651/1274] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/users => generated 291 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.713371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c57d51-a3c0-46bd-b8ab-5bdb6a19d75f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.713928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4986615b-0cf1-4cd7-88ae-8a912dd4f4b4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.725926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c57d51-a3c0-46bd-b8ab-5bdb6a19d75f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.728976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 624/1275] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:43 2022] PUT /identity/v3/projects/25f1ebbfb9d042fab037dbe07a651913/users/701e4507214f4634ad09e32e21749a35/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:43.729827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4986615b-0cf1-4cd7-88ae-8a912dd4f4b4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.746183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:43.787932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4707cdae-28fe-46dc-b05e-5a5fa7cfa696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.788598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4707cdae-28fe-46dc-b05e-5a5fa7cfa696 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.788880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4707cdae-28fe-46dc-b05e-5a5fa7cfa696 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.789156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4707cdae-28fe-46dc-b05e-5a5fa7cfa696 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.789793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4707cdae-28fe-46dc-b05e-5a5fa7cfa696 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.791725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4707cdae-28fe-46dc-b05e-5a5fa7cfa696 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.796023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4986615b-0cf1-4cd7-88ae-8a912dd4f4b4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.801626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a14017d-d506-45f0-8911-0e9894c21d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.802412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a14017d-d506-45f0-8911-0e9894c21d84 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.802533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a14017d-d506-45f0-8911-0e9894c21d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.802761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a14017d-d506-45f0-8911-0e9894c21d84 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.803407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a14017d-d506-45f0-8911-0e9894c21d84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.805047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a14017d-d506-45f0-8911-0e9894c21d84 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.819230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 625/1276] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.819230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a14017d-d506-45f0-8911-0e9894c21d84 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.819537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 652/1277] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/users => generated 294 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:43.819984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4707cdae-28fe-46dc-b05e-5a5fa7cfa696 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.833396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4707cdae-28fe-46dc-b05e-5a5fa7cfa696 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.833660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a14017d-d506-45f0-8911-0e9894c21d84 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.839167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce38061-07f9-46ec-a5f8-63436aee21ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.839751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85540fac-2bfe-454d-baa2-cf6ca8bb2904 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.840079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce38061-07f9-46ec-a5f8-63436aee21ea admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.840615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce38061-07f9-46ec-a5f8-63436aee21ea admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.840731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85540fac-2bfe-454d-baa2-cf6ca8bb2904 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.841068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce38061-07f9-46ec-a5f8-63436aee21ea admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.841171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85540fac-2bfe-454d-baa2-cf6ca8bb2904 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.841546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85540fac-2bfe-454d-baa2-cf6ca8bb2904 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.842116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce38061-07f9-46ec-a5f8-63436aee21ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.842599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85540fac-2bfe-454d-baa2-cf6ca8bb2904 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.845062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce38061-07f9-46ec-a5f8-63436aee21ea admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.845239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85540fac-2bfe-454d-baa2-cf6ca8bb2904 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.857969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 626/1278] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.857969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85540fac-2bfe-454d-baa2-cf6ca8bb2904 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.864218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 653/1279] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.865454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce38061-07f9-46ec-a5f8-63436aee21ea admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.871615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85540fac-2bfe-454d-baa2-cf6ca8bb2904 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.875760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce38061-07f9-46ec-a5f8-63436aee21ea admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.876214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a969d033-3c73-4385-b471-6a6599aab12b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.876825 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a969d033-3c73-4385-b471-6a6599aab12b admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.877121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a969d033-3c73-4385-b471-6a6599aab12b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.877368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a969d033-3c73-4385-b471-6a6599aab12b admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.878054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a969d033-3c73-4385-b471-6a6599aab12b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.879602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a969d033-3c73-4385-b471-6a6599aab12b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.880446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec76447-ea13-48a2-8263-ec8074f0073a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.881290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec76447-ea13-48a2-8263-ec8074f0073a admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.881569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec76447-ea13-48a2-8263-ec8074f0073a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.882209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec76447-ea13-48a2-8263-ec8074f0073a admin admin] PATH_INFO: `/v3/projects/fd4b77e78f9d4286aa92f6f90ff8b797/users/7127613bde784330bec364f02180291e/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.890750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 627/1280] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.891648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a969d033-3c73-4385-b471-6a6599aab12b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.901000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec76447-ea13-48a2-8263-ec8074f0073a admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd4b77e78f9d4286aa92f6f90ff8b797, user_id=7127613bde784330bec364f02180291e, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.901910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a969d033-3c73-4385-b471-6a6599aab12b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.903723 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec76447-ea13-48a2-8263-ec8074f0073a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.906501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b903a-4dee-43f1-afd4-f745b7d97eb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:43.907086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b903a-4dee-43f1-afd4-f745b7d97eb7 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.907404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b903a-4dee-43f1-afd4-f745b7d97eb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.907623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b903a-4dee-43f1-afd4-f745b7d97eb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.908857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870b903a-4dee-43f1-afd4-f745b7d97eb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:43.911198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870b903a-4dee-43f1-afd4-f745b7d97eb7 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:43.918378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 654/1281] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:43 2022] PUT /identity/v3/projects/fd4b77e78f9d4286aa92f6f90ff8b797/users/7127613bde784330bec364f02180291e/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:43.919885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be91c0-a376-48e4-becf-fd764a9d7590 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:43.920205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be91c0-a376-48e4-becf-fd764a9d7590 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:43.920466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be91c0-a376-48e4-becf-fd764a9d7590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:43.926130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 628/1282] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:43.926130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b903a-4dee-43f1-afd4-f745b7d97eb7 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:43.934112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b903a-4dee-43f1-afd4-f745b7d97eb7 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:43.943149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-20be91c0-a376-48e4-becf-fd764a9d7590 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:43.944377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-20be91c0-a376-48e4-becf-fd764a9d7590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:44.001526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2fa028-dcd6-4ad5-857e-1a8b4aa62be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.002485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2fa028-dcd6-4ad5-857e-1a8b4aa62be4 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.002752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2fa028-dcd6-4ad5-857e-1a8b4aa62be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.003029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2fa028-dcd6-4ad5-857e-1a8b4aa62be4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.003682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2fa028-dcd6-4ad5-857e-1a8b4aa62be4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.005385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2fa028-dcd6-4ad5-857e-1a8b4aa62be4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.013643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 655/1283] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:44.018572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 629/1284] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:44.018572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2fa028-dcd6-4ad5-857e-1a8b4aa62be4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.019389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20be91c0-a376-48e4-becf-fd764a9d7590 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.027505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20be91c0-a376-48e4-becf-fd764a9d7590 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.029888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2fa028-dcd6-4ad5-857e-1a8b4aa62be4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.037013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be6d018-1368-4542-8c25-b3a83e4cb473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.037591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba77e2f-a49b-45fa-85b7-0f9567ff8fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.037771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6d018-1368-4542-8c25-b3a83e4cb473 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.037956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6d018-1368-4542-8c25-b3a83e4cb473 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.038200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba77e2f-a49b-45fa-85b7-0f9567ff8fb7 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.038296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6d018-1368-4542-8c25-b3a83e4cb473 admin admin] PATH_INFO: `/v3/projects/2b6daac74a0c422794e65cfdf3bef30d/users/e41f9ffdfc3d4771a917fb25b5417a11/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.038503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba77e2f-a49b-45fa-85b7-0f9567ff8fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.038776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba77e2f-a49b-45fa-85b7-0f9567ff8fb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.039418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba77e2f-a49b-45fa-85b7-0f9567ff8fb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.040180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be6d018-1368-4542-8c25-b3a83e4cb473 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b6daac74a0c422794e65cfdf3bef30d, user_id=e41f9ffdfc3d4771a917fb25b5417a11, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.041110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba77e2f-a49b-45fa-85b7-0f9567ff8fb7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.042806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be6d018-1368-4542-8c25-b3a83e4cb473 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.056380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 656/1285] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:44 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:44.056578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 630/1286] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:44 2022] PUT /identity/v3/projects/2b6daac74a0c422794e65cfdf3bef30d/users/e41f9ffdfc3d4771a917fb25b5417a11/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:44.056664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba77e2f-a49b-45fa-85b7-0f9567ff8fb7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.060765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be6d018-1368-4542-8c25-b3a83e4cb473 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.065625 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba77e2f-a49b-45fa-85b7-0f9567ff8fb7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.076650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:44.124602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4aa0a8-c66e-4a4e-a44a-7c607e6e9705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.125211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4aa0a8-c66e-4a4e-a44a-7c607e6e9705 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.125498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4aa0a8-c66e-4a4e-a44a-7c607e6e9705 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.125769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4aa0a8-c66e-4a4e-a44a-7c607e6e9705 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.126403 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4aa0a8-c66e-4a4e-a44a-7c607e6e9705 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.128240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4aa0a8-c66e-4a4e-a44a-7c607e6e9705 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.130788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be6d018-1368-4542-8c25-b3a83e4cb473 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.135895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d638200e-f114-4f6d-9e36-2b886d97521e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.136599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d638200e-f114-4f6d-9e36-2b886d97521e admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.136876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d638200e-f114-4f6d-9e36-2b886d97521e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.137147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d638200e-f114-4f6d-9e36-2b886d97521e admin admin] PATH_INFO: `/v3/projects/0216a8024d3146799f1ecf71ee4de0bd/users/0f9689838e2d45039b78ddac79d77df6/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.138758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 657/1287] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:44 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:44.140020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aa2111-ef6d-469c-8fed-47941660bcc9 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.140332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aa2111-ef6d-469c-8fed-47941660bcc9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.141454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aa2111-ef6d-469c-8fed-47941660bcc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.148178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:44.154492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d638200e-f114-4f6d-9e36-2b886d97521e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0216a8024d3146799f1ecf71ee4de0bd, user_id=0f9689838e2d45039b78ddac79d77df6, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.158638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d638200e-f114-4f6d-9e36-2b886d97521e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.163913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f0aa2111-ef6d-469c-8fed-47941660bcc9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:44.165365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f0aa2111-ef6d-469c-8fed-47941660bcc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:44.172025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 631/1288] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:44 2022] PUT /identity/v3/projects/0216a8024d3146799f1ecf71ee4de0bd/users/0f9689838e2d45039b78ddac79d77df6/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:44.173865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d638200e-f114-4f6d-9e36-2b886d97521e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.240618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 658/1289] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:44.244787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d638200e-f114-4f6d-9e36-2b886d97521e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.245201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aa2111-ef6d-469c-8fed-47941660bcc9 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.249518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa930be-ec25-4262-8223-aecb31cf5347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.250046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa930be-ec25-4262-8223-aecb31cf5347 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.250325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa930be-ec25-4262-8223-aecb31cf5347 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.250602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa930be-ec25-4262-8223-aecb31cf5347 admin admin] PATH_INFO: `/v3/projects/9ba8c5dfb3d04ff098905bd7841ad64b/users/b98ba8906d524e099f6f4dd1871871ea/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.255472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aa2111-ef6d-469c-8fed-47941660bcc9 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.260204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbd470e-0c4b-45de-829e-00e1df2dc0c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.260882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbd470e-0c4b-45de-829e-00e1df2dc0c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.261149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbd470e-0c4b-45de-829e-00e1df2dc0c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.261424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbd470e-0c4b-45de-829e-00e1df2dc0c2 admin admin] PATH_INFO: `/v3/projects/2b2bf942072c4ef684069f485877d379/users/6661da2c7e754749a15bf615c23fb933/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.265245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa930be-ec25-4262-8223-aecb31cf5347 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ba8c5dfb3d04ff098905bd7841ad64b, user_id=b98ba8906d524e099f6f4dd1871871ea, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.268467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa930be-ec25-4262-8223-aecb31cf5347 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.278523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbd470e-0c4b-45de-829e-00e1df2dc0c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b2bf942072c4ef684069f485877d379, user_id=6661da2c7e754749a15bf615c23fb933, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.281218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbd470e-0c4b-45de-829e-00e1df2dc0c2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.284393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 632/1290] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:44 2022] PUT /identity/v3/projects/9ba8c5dfb3d04ff098905bd7841ad64b/users/b98ba8906d524e099f6f4dd1871871ea/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:44.285466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa930be-ec25-4262-8223-aecb31cf5347 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.296107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 659/1291] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:44 2022] PUT /identity/v3/projects/2b2bf942072c4ef684069f485877d379/users/6661da2c7e754749a15bf615c23fb933/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:44.296984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbd470e-0c4b-45de-829e-00e1df2dc0c2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.343473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa930be-ec25-4262-8223-aecb31cf5347 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.368029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbd470e-0c4b-45de-829e-00e1df2dc0c2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.372845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a89f44f-9fd8-4c71-8ce6-ffcf6658e396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.373494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a89f44f-9fd8-4c71-8ce6-ffcf6658e396 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.373828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a89f44f-9fd8-4c71-8ce6-ffcf6658e396 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.374143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a89f44f-9fd8-4c71-8ce6-ffcf6658e396 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.374822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a89f44f-9fd8-4c71-8ce6-ffcf6658e396 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.375350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:44.376610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a89f44f-9fd8-4c71-8ce6-ffcf6658e396 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.387663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 660/1292] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:44 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:44.388928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453c9c5c-a548-4140-8b8f-526d5d7edd70 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.389224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453c9c5c-a548-4140-8b8f-526d5d7edd70 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.389484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453c9c5c-a548-4140-8b8f-526d5d7edd70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.400652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d7aa7-97e1-407e-85b2-bc562ad82f88 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.401274 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d7aa7-97e1-407e-85b2-bc562ad82f88 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.401540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d7aa7-97e1-407e-85b2-bc562ad82f88 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.401815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d7aa7-97e1-407e-85b2-bc562ad82f88 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.412077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2d7aa7-97e1-407e-85b2-bc562ad82f88 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.413584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2d7aa7-97e1-407e-85b2-bc562ad82f88 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.423376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-453c9c5c-a548-4140-8b8f-526d5d7edd70 None None] MFA Rules not processed for user `e41f9ffdfc3d4771a917fb25b5417a11`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:44.424935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 633/1293] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:44.425443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-453c9c5c-a548-4140-8b8f-526d5d7edd70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:44.425678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d7aa7-97e1-407e-85b2-bc562ad82f88 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.435329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d7aa7-97e1-407e-85b2-bc562ad82f88 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.440780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec794fc-5ba3-4489-8cc0-14856b499fdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.442576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec794fc-5ba3-4489-8cc0-14856b499fdb admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.442734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec794fc-5ba3-4489-8cc0-14856b499fdb admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.442977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec794fc-5ba3-4489-8cc0-14856b499fdb admin admin] PATH_INFO: `/v3/projects/b34f10ec378a4622a8f0b0546e84da7f/users/ebdef1d69bb14837b9a0c28cbba332fc/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.469126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec794fc-5ba3-4489-8cc0-14856b499fdb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b34f10ec378a4622a8f0b0546e84da7f, user_id=ebdef1d69bb14837b9a0c28cbba332fc, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.471245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec794fc-5ba3-4489-8cc0-14856b499fdb admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.489528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 634/1294] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:44 2022] PUT /identity/v3/projects/b34f10ec378a4622a8f0b0546e84da7f/users/ebdef1d69bb14837b9a0c28cbba332fc/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:44.490498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28f571b-89ec-498d-b8e3-52792c5f25dd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.490791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28f571b-89ec-498d-b8e3-52792c5f25dd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.491359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28f571b-89ec-498d-b8e3-52792c5f25dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.517406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:44.521894 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c28f571b-89ec-498d-b8e3-52792c5f25dd None None] MFA Rules not processed for user `0f9689838e2d45039b78ddac79d77df6`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:44.523158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c28f571b-89ec-498d-b8e3-52792c5f25dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:44.524269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 661/1295] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:44 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:44.525771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453c9c5c-a548-4140-8b8f-526d5d7edd70 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.547232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453c9c5c-a548-4140-8b8f-526d5d7edd70 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.607335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 635/1296] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:44 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:44.608086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28f571b-89ec-498d-b8e3-52792c5f25dd None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.611945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78bafef-e547-488e-aa2b-43138c65f276 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.612423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78bafef-e547-488e-aa2b-43138c65f276 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.612704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78bafef-e547-488e-aa2b-43138c65f276 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.612977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78bafef-e547-488e-aa2b-43138c65f276 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.687803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28f571b-89ec-498d-b8e3-52792c5f25dd None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.692673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691b530d-eab4-450f-abfe-01df5d1d1ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.693307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b530d-eab4-450f-abfe-01df5d1d1ce6 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.693753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b530d-eab4-450f-abfe-01df5d1d1ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.694041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b530d-eab4-450f-abfe-01df5d1d1ce6 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.694693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691b530d-eab4-450f-abfe-01df5d1d1ce6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.695711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78bafef-e547-488e-aa2b-43138c65f276 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.696403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691b530d-eab4-450f-abfe-01df5d1d1ce6 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.697374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78bafef-e547-488e-aa2b-43138c65f276 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.711160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 636/1297] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:44 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:44.711160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6a0a7b-ede7-48c8-a9e0-d8ac833c7391 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.711160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6a0a7b-ede7-48c8-a9e0-d8ac833c7391 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.711160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6a0a7b-ede7-48c8-a9e0-d8ac833c7391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.711956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 662/1298] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:44.711956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78bafef-e547-488e-aa2b-43138c65f276 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.723383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78bafef-e547-488e-aa2b-43138c65f276 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.729294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f0cf9b-0341-460b-91b6-55e432cfdf2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.730301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f0cf9b-0341-460b-91b6-55e432cfdf2d admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.730778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f0cf9b-0341-460b-91b6-55e432cfdf2d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.730917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f0cf9b-0341-460b-91b6-55e432cfdf2d admin admin] PATH_INFO: `/v3/projects/7ff48b26c576495f8bb8c88d20d246cc/users/552c23b8456a40ec8e01297aca7965d3/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.743964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9e6a0a7b-ede7-48c8-a9e0-d8ac833c7391 None None] MFA Rules not processed for user `6661da2c7e754749a15bf615c23fb933`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:44.745073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9e6a0a7b-ede7-48c8-a9e0-d8ac833c7391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:44.757150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f0cf9b-0341-460b-91b6-55e432cfdf2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ff48b26c576495f8bb8c88d20d246cc, user_id=552c23b8456a40ec8e01297aca7965d3, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.761161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f0cf9b-0341-460b-91b6-55e432cfdf2d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:44.780173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 663/1299] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:44 2022] PUT /identity/v3/projects/7ff48b26c576495f8bb8c88d20d246cc/users/552c23b8456a40ec8e01297aca7965d3/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:44.782071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0193fd-3c1c-42c5-844a-b50d90109ec8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.782467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0193fd-3c1c-42c5-844a-b50d90109ec8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.782816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0193fd-3c1c-42c5-844a-b50d90109ec8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.794596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:44.816502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ce0193fd-3c1c-42c5-844a-b50d90109ec8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:44.818197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ce0193fd-3c1c-42c5-844a-b50d90109ec8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:44.895622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 637/1300] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:44 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:44.896966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6a0a7b-ede7-48c8-a9e0-d8ac833c7391 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.947685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 664/1301] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:44.947685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0193fd-3c1c-42c5-844a-b50d90109ec8 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:44.957603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0193fd-3c1c-42c5-844a-b50d90109ec8 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:44.970655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ba607f-c0a0-419f-a961-e9e26098b04a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:44.971401 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba607f-c0a0-419f-a961-e9e26098b04a admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:44.971774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba607f-c0a0-419f-a961-e9e26098b04a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:44.972097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba607f-c0a0-419f-a961-e9e26098b04a admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:44.972799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ba607f-c0a0-419f-a961-e9e26098b04a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:44.976426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ba607f-c0a0-419f-a961-e9e26098b04a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.010296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6a0a7b-ede7-48c8-a9e0-d8ac833c7391 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:45.036926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 665/1302] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:44 2022] POST /identity/v3/projects => generated 357 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:45.039932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ba607f-c0a0-419f-a961-e9e26098b04a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:45.047538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:45.109620 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134fd644-bbdf-4589-b101-21a108ff4970 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:45.110241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134fd644-bbdf-4589-b101-21a108ff4970 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.110519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134fd644-bbdf-4589-b101-21a108ff4970 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.110789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134fd644-bbdf-4589-b101-21a108ff4970 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.137314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ba607f-c0a0-419f-a961-e9e26098b04a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:45.143818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edfdf15-ed11-49b8-a550-14bf1917c9af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:45.144550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edfdf15-ed11-49b8-a550-14bf1917c9af admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.144875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edfdf15-ed11-49b8-a550-14bf1917c9af admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.145156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edfdf15-ed11-49b8-a550-14bf1917c9af admin admin] PATH_INFO: `/v3/projects/9ba8c5dfb3d04ff098905bd7841ad64b/users/b98ba8906d524e099f6f4dd1871871ea/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.147028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edfdf15-ed11-49b8-a550-14bf1917c9af admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ba8c5dfb3d04ff098905bd7841ad64b, user_id=b98ba8906d524e099f6f4dd1871871ea, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:45.149995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edfdf15-ed11-49b8-a550-14bf1917c9af admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.166260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 666/1303] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:45 2022] PUT /identity/v3/projects/9ba8c5dfb3d04ff098905bd7841ad64b/users/b98ba8906d524e099f6f4dd1871871ea/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:45.167944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494c88d6-4bcb-470e-a104-46afdf775c25 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.168174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494c88d6-4bcb-470e-a104-46afdf775c25 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.168456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494c88d6-4bcb-470e-a104-46afdf775c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.195690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-494c88d6-4bcb-470e-a104-46afdf775c25 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:45.197165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-494c88d6-4bcb-470e-a104-46afdf775c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:45.264940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134fd644-bbdf-4589-b101-21a108ff4970 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:45.269219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134fd644-bbdf-4589-b101-21a108ff4970 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.291928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 638/1304] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:44 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:45.294884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134fd644-bbdf-4589-b101-21a108ff4970 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:45.308337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134fd644-bbdf-4589-b101-21a108ff4970 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:45.343204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 667/1305] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:45.347847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494c88d6-4bcb-470e-a104-46afdf775c25 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:45.399059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:45.451972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832914c3-3f09-4022-a84f-cef28fd74b5b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:45.452566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832914c3-3f09-4022-a84f-cef28fd74b5b service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.452851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832914c3-3f09-4022-a84f-cef28fd74b5b service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.453338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832914c3-3f09-4022-a84f-cef28fd74b5b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.520497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494c88d6-4bcb-470e-a104-46afdf775c25 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:45.533894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a36d06-4e18-409d-be41-540d96bfdc79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:45.537189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a36d06-4e18-409d-be41-540d96bfdc79 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.537189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a36d06-4e18-409d-be41-540d96bfdc79 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.540184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a36d06-4e18-409d-be41-540d96bfdc79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.561244 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a36d06-4e18-409d-be41-540d96bfdc79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:45.563082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a36d06-4e18-409d-be41-540d96bfdc79 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.587199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 668/1306] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:45.589525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832914c3-3f09-4022-a84f-cef28fd74b5b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:45.591258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a36d06-4e18-409d-be41-540d96bfdc79 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:45.591417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832914c3-3f09-4022-a84f-cef28fd74b5b service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.604318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a36d06-4e18-409d-be41-540d96bfdc79 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:45.609819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 639/1307] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:45 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:45.610577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1266a26-9d42-45bd-8414-fb84a1736ffa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:45.612259 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1266a26-9d42-45bd-8414-fb84a1736ffa admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.612727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1266a26-9d42-45bd-8414-fb84a1736ffa admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.613369 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1266a26-9d42-45bd-8414-fb84a1736ffa admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.614180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1266a26-9d42-45bd-8414-fb84a1736ffa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:45.615306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63224e1-ff98-4877-b31f-b8eb3019e0b4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.617165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1266a26-9d42-45bd-8414-fb84a1736ffa admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.618749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63224e1-ff98-4877-b31f-b8eb3019e0b4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.618910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63224e1-ff98-4877-b31f-b8eb3019e0b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.664761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 669/1308] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:45 2022] POST /identity/v3/users => generated 285 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:45.664761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1266a26-9d42-45bd-8414-fb84a1736ffa admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:45.680111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1266a26-9d42-45bd-8414-fb84a1736ffa admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:45.686375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf6711e-ac39-4d99-bcc3-4b79ab6cc0e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:45.687207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6711e-ac39-4d99-bcc3-4b79ab6cc0e3 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.687682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6711e-ac39-4d99-bcc3-4b79ab6cc0e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.688085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6711e-ac39-4d99-bcc3-4b79ab6cc0e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.691710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f63224e1-ff98-4877-b31f-b8eb3019e0b4 None None] MFA Rules not processed for user `b98ba8906d524e099f6f4dd1871871ea`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:45.693052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f63224e1-ff98-4877-b31f-b8eb3019e0b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:45.702701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf6711e-ac39-4d99-bcc3-4b79ab6cc0e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:45.704500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf6711e-ac39-4d99-bcc3-4b79ab6cc0e3 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.720608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 670/1309] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:45.721874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf6711e-ac39-4d99-bcc3-4b79ab6cc0e3 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:45.750900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf6711e-ac39-4d99-bcc3-4b79ab6cc0e3 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:45.772040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:45.773374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.774230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.774230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] PATH_INFO: `/v3/users/f39e25b9275e431f8a7304171a0c1ffb` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.782429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f39e25b9275e431f8a7304171a0c1ffb)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:45.785310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: WARNING py.warnings [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:50:45.785310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: warnings.warn(msg) Oct 24 08:50:45.785310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: Oct 24 08:50:45.788724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.811596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:45.905446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 640/1310] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:45 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:45.913125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63224e1-ff98-4877-b31f-b8eb3019e0b4 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:45.937498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63224e1-ff98-4877-b31f-b8eb3019e0b4 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:45.951423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3a5f57-590a-4684-a9f9-87d22f062812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:45.952386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a5f57-590a-4684-a9f9-87d22f062812 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:45.952948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a5f57-590a-4684-a9f9-87d22f062812 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:45.953389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a5f57-590a-4684-a9f9-87d22f062812 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:45.954345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3a5f57-590a-4684-a9f9-87d22f062812 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:45.955695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f39e25b9275e431f8a7304171a0c1ffb'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:45.956543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3a5f57-590a-4684-a9f9-87d22f062812 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:45.977098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f39e25b9275e431f8a7304171a0c1ffb'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:45.981293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 641/1311] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:45 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:45.982499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3a5f57-590a-4684-a9f9-87d22f062812 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:46.005900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3a5f57-590a-4684-a9f9-87d22f062812 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:46.016339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f39e25b9275e431f8a7304171a0c1ffb'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:46.016823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d379f596-a298-4349-9e1d-5fbda02467d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:46.017429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f39e25b9275e431f8a7304171a0c1ffb'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:46.017882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d379f596-a298-4349-9e1d-5fbda02467d9 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:46.018346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d379f596-a298-4349-9e1d-5fbda02467d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:46.018684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f39e25b9275e431f8a7304171a0c1ffb'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:46.018940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d379f596-a298-4349-9e1d-5fbda02467d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:46.025050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d379f596-a298-4349-9e1d-5fbda02467d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:46.044066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 671/1312] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:50:45 2022] DELETE /identity/v3/users/f39e25b9275e431f8a7304171a0c1ffb => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:46.047269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:46.047566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:46.054727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d379f596-a298-4349-9e1d-5fbda02467d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:46.056576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d379f596-a298-4349-9e1d-5fbda02467d9 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:46.078041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 642/1313] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:45 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:46.080622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbf6de5-fe50-4adf-83ea-d64e78c61ce6 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:46.081258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbf6de5-fe50-4adf-83ea-d64e78c61ce6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:46.081708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbf6de5-fe50-4adf-83ea-d64e78c61ce6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:46.111974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2fbf6de5-fe50-4adf-83ea-d64e78c61ce6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:46.113423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2fbf6de5-fe50-4adf-83ea-d64e78c61ce6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:46.192330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:46.237319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 643/1314] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:46.244284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8884ba16-c855-48c3-9d9b-7ec85bf699b4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:46.252875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:46.253808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:46.254248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:46.254685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] PATH_INFO: `/v3/projects/d3cdd94a225a46a8bddd2fcfe7d42430/users/d489a074003e4f70acba7f849fbd839b/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:46.273730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3cdd94a225a46a8bddd2fcfe7d42430, user_id=d489a074003e4f70acba7f849fbd839b, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:46.278061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:46.304277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 672/1315] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:46 2022] PUT /identity/v3/projects/d3cdd94a225a46a8bddd2fcfe7d42430/users/d489a074003e4f70acba7f849fbd839b/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:46.308634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbf6de5-fe50-4adf-83ea-d64e78c61ce6 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:46.324462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbf6de5-fe50-4adf-83ea-d64e78c61ce6 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:46.345077 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:46.351420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:46.508123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:46.524164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:46.524164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:46.524164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:46.524164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:46.524164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:46.540527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c31abe-0c2e-4f22-a223-5eb880819067 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:46.550503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 644/1316] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:46 2022] GET /identity/v3/roles => generated 1420 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:46.551600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:46.553257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:46.554017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:46.555204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:46.555513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:46.558878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:46.559344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:46.613808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:46.716118 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bb5085-f582-4ad0-a38d-556406859d5c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:46.740861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:46.743546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d9a422-e612-4f59-8339-7825c5764ebc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:46.744385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d9a422-e612-4f59-8339-7825c5764ebc service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:46.744673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d9a422-e612-4f59-8339-7825c5764ebc service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:46.744950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d9a422-e612-4f59-8339-7825c5764ebc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:46.746427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:46.788820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 673/1317] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:46 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:50:46.791423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:46.792239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d9a422-e612-4f59-8339-7825c5764ebc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:46.797257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d9a422-e612-4f59-8339-7825c5764ebc service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:46.835140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91da0a1b-9c25-486c-99e4-89b8445dd4bc service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:46.839959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 645/1318] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:46.850903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:46.851602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:46.851885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:46.852182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] PATH_INFO: `/v3/projects/d3cdd94a225a46a8bddd2fcfe7d42430/users/d489a074003e4f70acba7f849fbd839b/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:46.858186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3cdd94a225a46a8bddd2fcfe7d42430, user_id=d489a074003e4f70acba7f849fbd839b, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:46.861776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:46.892575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 674/1319] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:46 2022] PUT /identity/v3/projects/d3cdd94a225a46a8bddd2fcfe7d42430/users/d489a074003e4f70acba7f849fbd839b/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:46.904120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9dc200-923e-4e39-964d-8382c8a2d8dc None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:46.904906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9dc200-923e-4e39-964d-8382c8a2d8dc None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:46.905443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9dc200-923e-4e39-964d-8382c8a2d8dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:46.977008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-8d9dc200-923e-4e39-964d-8382c8a2d8dc None None] MFA Rules not processed for user `d489a074003e4f70acba7f849fbd839b`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:46.983018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-8d9dc200-923e-4e39-964d-8382c8a2d8dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:47.196787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 646/1320] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:46 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:47.209998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:47.211633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:47.373129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bed49e-3c04-4ae0-9331-0a928dbcc44c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:47.379448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:47.380420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:47.380742 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:47.381016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:47.395125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:47.397179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:47.412202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 675/1321] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:47 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:47.545385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9dc200-923e-4e39-964d-8382c8a2d8dc None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:47.559291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9dc200-923e-4e39-964d-8382c8a2d8dc None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:47.563686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:47.564881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:47.568667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:47.568667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:47.568667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:47.583391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:47.656795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:47.658703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:47.672450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 647/1322] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:47 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:48.171249 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:48.173304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:48.365903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab56a125-24c3-4629-8731-42b6ce49a43b service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:48.380909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:48.382554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:48.382845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:48.383122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] PATH_INFO: `/v3/projects/246835f30eb64b17ac0adccb40379b7f` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:48.386413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] RBAC: Authorizing `identity:delete_project(project_id=246835f30eb64b17ac0adccb40379b7f)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:48.391178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:48.526353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '246835f30eb64b17ac0adccb40379b7f', 'request_id': 'req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:48.531341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:48.556641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '246835f30eb64b17ac0adccb40379b7f', 'request_id': 'req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:48.574866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '246835f30eb64b17ac0adccb40379b7f', 'request_id': 'req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:48.597693 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] The token cache is being invalidate because project 246835f30eb64b17ac0adccb40379b7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:50:48.597886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:48.598573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a06bd41b-83d0-4ad2-b2f4-8c04250dc099 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:48.604609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 676/1323] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:50:48 2022] DELETE /identity/v3/projects/246835f30eb64b17ac0adccb40379b7f => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:48.609006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:48.629187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a4f2d5-d500-4814-9980-ed1d32642ef0 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:48.777602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:48.778776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:48.779254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:48.779535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] PATH_INFO: `/v3/users/e41f9ffdfc3d4771a917fb25b5417a11` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:48.780842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e41f9ffdfc3d4771a917fb25b5417a11)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:48.786103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:48.968076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e41f9ffdfc3d4771a917fb25b5417a11'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:48.992444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e41f9ffdfc3d4771a917fb25b5417a11'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:49.051679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e41f9ffdfc3d4771a917fb25b5417a11'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:49.057308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e41f9ffdfc3d4771a917fb25b5417a11'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:49.059140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f45bc3e5-f91e-43e3-a37f-2e6c6ef7b8c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e41f9ffdfc3d4771a917fb25b5417a11'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:49.106694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 648/1324] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:50:48 2022] DELETE /identity/v3/users/e41f9ffdfc3d4771a917fb25b5417a11 => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:49.922261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae59fc29-37ff-468c-977f-9003698e36fe None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:49.923171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae59fc29-37ff-468c-977f-9003698e36fe None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:49.923637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae59fc29-37ff-468c-977f-9003698e36fe None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:49.925683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 677/1325] 10.222.0.39 () {54 vars in 961 bytes} [Mon Oct 24 08:50:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:50:49.943938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c838ad6c-3cdd-4fdc-9c25-cca69cb870ca None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:49.944401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c838ad6c-3cdd-4fdc-9c25-cca69cb870ca None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:49.944615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c838ad6c-3cdd-4fdc-9c25-cca69cb870ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:50.052376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c838ad6c-3cdd-4fdc-9c25-cca69cb870ca None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:50.054292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c838ad6c-3cdd-4fdc-9c25-cca69cb870ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:50.085935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:50.229756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 649/1326] 10.222.0.39 () {58 vars in 1058 bytes} [Mon Oct 24 08:50:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Oct 24 08:50:50.238372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a3f23-22fd-485d-888c-3ee4b6d11c7f None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:50.238914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a3f23-22fd-485d-888c-3ee4b6d11c7f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:50.239408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a3f23-22fd-485d-888c-3ee4b6d11c7f None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:50.241330 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 678/1327] 10.222.0.39 () {54 vars in 912 bytes} [Mon Oct 24 08:50:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:50:50.247716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e442d5b-5ad0-4531-a820-8f0dc098f543 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:50.248581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e442d5b-5ad0-4531-a820-8f0dc098f543 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:50.248581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e442d5b-5ad0-4531-a820-8f0dc098f543 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:50.287742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7e442d5b-5ad0-4531-a820-8f0dc098f543 None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:50.292055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7e442d5b-5ad0-4531-a820-8f0dc098f543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:50.316027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 650/1328] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:50:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:50.334254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92a3f23-22fd-485d-888c-3ee4b6d11c7f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:50.336404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a92a3f23-22fd-485d-888c-3ee4b6d11c7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:50.522719 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92a3f23-22fd-485d-888c-3ee4b6d11c7f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:50.533794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e662c21-d193-4809-a09d-30dc627d96b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:50.534650 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e662c21-d193-4809-a09d-30dc627d96b9 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:50.534936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e662c21-d193-4809-a09d-30dc627d96b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:50.535297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e662c21-d193-4809-a09d-30dc627d96b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:50.562179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e662c21-d193-4809-a09d-30dc627d96b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:50.568646 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e662c21-d193-4809-a09d-30dc627d96b9 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:50.592204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 679/1329] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:50.697305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e442d5b-5ad0-4531-a820-8f0dc098f543 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:50.699454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7e442d5b-5ad0-4531-a820-8f0dc098f543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:50.950410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e442d5b-5ad0-4531-a820-8f0dc098f543 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:50.965540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:50.966359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:50.966908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:50.967370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] PATH_INFO: `/v3/projects/2b6daac74a0c422794e65cfdf3bef30d` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:50.971532 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b6daac74a0c422794e65cfdf3bef30d)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:50.972974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: WARNING py.warnings [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 24 08:50:50.972974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: warnings.warn(msg) Oct 24 08:50:50.972974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: Oct 24 08:50:50.974676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:51.166819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b6daac74a0c422794e65cfdf3bef30d', 'request_id': 'req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:51.178456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b6daac74a0c422794e65cfdf3bef30d', 'request_id': 'req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:51.236658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b6daac74a0c422794e65cfdf3bef30d', 'request_id': 'req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:51.272767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] The token cache is being invalidate because project 2b6daac74a0c422794e65cfdf3bef30d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:50:51.273605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:51.277306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3fcde9a7-ea61-4b3b-a947-9828ef69b33c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:50:51.282517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 651/1330] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:50:50 2022] DELETE /identity/v3/projects/2b6daac74a0c422794e65cfdf3bef30d => generated 0 bytes in 582 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:51.285401 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3e353-2367-4508-9316-2186397b9960 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:51.285907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3e353-2367-4508-9316-2186397b9960 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:51.286010 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3e353-2367-4508-9316-2186397b9960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:51.334814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b0a3e353-2367-4508-9316-2186397b9960 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:51.336040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b0a3e353-2367-4508-9316-2186397b9960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:51.550346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 680/1331] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:51.559623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc3f1c-41ef-4a85-9865-853a09c19e04 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:51.559912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc3f1c-41ef-4a85-9865-853a09c19e04 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:51.560246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc3f1c-41ef-4a85-9865-853a09c19e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:51.612847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-26dc3f1c-41ef-4a85-9865-853a09c19e04 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:51.614163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-26dc3f1c-41ef-4a85-9865-853a09c19e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:51.638805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 652/1332] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:51.640884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a3e353-2367-4508-9316-2186397b9960 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:51.657007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a3e353-2367-4508-9316-2186397b9960 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:51.681869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:51.682833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:51.683253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:51.683836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:51.684669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:50:51.685137 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:51.687229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:51.741324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 681/1333] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:50:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:50:51.747183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dc3f1c-41ef-4a85-9865-853a09c19e04 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:51.764403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dc3f1c-41ef-4a85-9865-853a09c19e04 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:51.773830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae1b7d-cf00-431d-91e8-3bc6f9b2ea39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:51.774596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae1b7d-cf00-431d-91e8-3bc6f9b2ea39 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:51.774887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae1b7d-cf00-431d-91e8-3bc6f9b2ea39 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:51.775362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae1b7d-cf00-431d-91e8-3bc6f9b2ea39 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:51.776020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acae1b7d-cf00-431d-91e8-3bc6f9b2ea39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:51.781231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acae1b7d-cf00-431d-91e8-3bc6f9b2ea39 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:51.848692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 653/1334] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:51 2022] POST /identity/v3/projects => generated 369 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:51.850321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:51.866945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37021a1c-b0f5-4318-8787-4eff8dac1db6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:52.075034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1eb7e0-dd92-42a3-ad27-96c1846bc8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:52.075665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1eb7e0-dd92-42a3-ad27-96c1846bc8bd admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:52.075940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1eb7e0-dd92-42a3-ad27-96c1846bc8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:52.076241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1eb7e0-dd92-42a3-ad27-96c1846bc8bd admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:52.078173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1eb7e0-dd92-42a3-ad27-96c1846bc8bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:52.085266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1eb7e0-dd92-42a3-ad27-96c1846bc8bd admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:52.135240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:52.158873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 682/1335] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:51 2022] POST /identity/v3/users => generated 285 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:52.159580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae1b7d-cf00-431d-91e8-3bc6f9b2ea39 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:52.172970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:52.178867 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae1b7d-cf00-431d-91e8-3bc6f9b2ea39 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:52.191454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdcc0c0-48d7-4f5d-bb2d-7a685594bf8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:52.192597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdcc0c0-48d7-4f5d-bb2d-7a685594bf8a admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:52.193052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdcc0c0-48d7-4f5d-bb2d-7a685594bf8a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:52.193487 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdcc0c0-48d7-4f5d-bb2d-7a685594bf8a admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:52.194314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdcc0c0-48d7-4f5d-bb2d-7a685594bf8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:52.196206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdcc0c0-48d7-4f5d-bb2d-7a685594bf8a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:52.224810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 654/1336] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:52 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:50:52.229276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1eb7e0-dd92-42a3-ad27-96c1846bc8bd admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:52.248875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1eb7e0-dd92-42a3-ad27-96c1846bc8bd admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:52.262464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df5e3f1-b23a-4ca5-b782-a7788c2302f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:52.263364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5e3f1-b23a-4ca5-b782-a7788c2302f6 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:52.263831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5e3f1-b23a-4ca5-b782-a7788c2302f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:52.264312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5e3f1-b23a-4ca5-b782-a7788c2302f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:52.265506 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df5e3f1-b23a-4ca5-b782-a7788c2302f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:52.267412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df5e3f1-b23a-4ca5-b782-a7788c2302f6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:52.311576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdcc0c0-48d7-4f5d-bb2d-7a685594bf8a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:52.312644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 683/1337] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:52 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:52.333272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdcc0c0-48d7-4f5d-bb2d-7a685594bf8a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:52.344908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54ea083-fe4b-413f-b88a-3f1508ab9454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:52.346064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54ea083-fe4b-413f-b88a-3f1508ab9454 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:52.346252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54ea083-fe4b-413f-b88a-3f1508ab9454 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:52.348643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54ea083-fe4b-413f-b88a-3f1508ab9454 admin admin] PATH_INFO: `/v3/projects/e4d18888ab7a4325b341d6a2a63ef7d6/users/3bc80d8fab5a4f9bbdc605a7689f21fe/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:52.391702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54ea083-fe4b-413f-b88a-3f1508ab9454 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4d18888ab7a4325b341d6a2a63ef7d6, user_id=3bc80d8fab5a4f9bbdc605a7689f21fe, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:52.396941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54ea083-fe4b-413f-b88a-3f1508ab9454 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:52.425854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 655/1338] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:52 2022] PUT /identity/v3/projects/e4d18888ab7a4325b341d6a2a63ef7d6/users/3bc80d8fab5a4f9bbdc605a7689f21fe/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:50:52.437975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfc9898-6750-402b-880e-d30cb6e1c8ec None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:52.438348 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfc9898-6750-402b-880e-d30cb6e1c8ec None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:52.438633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfc9898-6750-402b-880e-d30cb6e1c8ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:52.542285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-bbfc9898-6750-402b-880e-d30cb6e1c8ec None None] MFA Rules not processed for user `3bc80d8fab5a4f9bbdc605a7689f21fe`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:52.560084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bbfc9898-6750-402b-880e-d30cb6e1c8ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:52.807563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 684/1339] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:52 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 371 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:52.812106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54ea083-fe4b-413f-b88a-3f1508ab9454 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:53.027714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54ea083-fe4b-413f-b88a-3f1508ab9454 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:53.052751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44826706-1f38-4406-a0c7-dffe14c3ca8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:53.053393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44826706-1f38-4406-a0c7-dffe14c3ca8e admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.053691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44826706-1f38-4406-a0c7-dffe14c3ca8e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.053958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44826706-1f38-4406-a0c7-dffe14c3ca8e admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.054617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44826706-1f38-4406-a0c7-dffe14c3ca8e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:53.059299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44826706-1f38-4406-a0c7-dffe14c3ca8e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:53.123475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b138b0-fdec-4d12-ab9c-146f1e923830 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.123821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b138b0-fdec-4d12-ab9c-146f1e923830 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.124039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b138b0-fdec-4d12-ab9c-146f1e923830 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.125703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 685/1340] 10.222.0.39 () {54 vars in 961 bytes} [Mon Oct 24 08:50:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:50:53.128923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 656/1341] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:50:52 2022] POST /identity/v3/projects => generated 369 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:50:53.132951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b138b0-fdec-4d12-ab9c-146f1e923830 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:53.153679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81f7478-0ddd-49d2-a61c-15754827d5dd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.154079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81f7478-0ddd-49d2-a61c-15754827d5dd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.154257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81f7478-0ddd-49d2-a61c-15754827d5dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.276035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a81f7478-0ddd-49d2-a61c-15754827d5dd None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:53.299960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a81f7478-0ddd-49d2-a61c-15754827d5dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:53.421919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b138b0-fdec-4d12-ab9c-146f1e923830 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:53.442698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a86ea-8e25-4007-a84c-6866e39d662f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:53.443341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a86ea-8e25-4007-a84c-6866e39d662f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.443625 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a86ea-8e25-4007-a84c-6866e39d662f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.443898 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a86ea-8e25-4007-a84c-6866e39d662f admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.444606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809a86ea-8e25-4007-a84c-6866e39d662f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:53.458033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809a86ea-8e25-4007-a84c-6866e39d662f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:53.550773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 686/1342] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:50:53 2022] POST /identity/v3/users => generated 291 bytes in 415 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:50:53.552937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 657/1343] 10.222.0.39 () {58 vars in 1058 bytes} [Mon Oct 24 08:50:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 400 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:50:53.558498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a86ea-8e25-4007-a84c-6866e39d662f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:53.558723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963ec45-13ac-41c7-b1ca-ba8bf39e1fd8 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.558723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963ec45-13ac-41c7-b1ca-ba8bf39e1fd8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.558897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963ec45-13ac-41c7-b1ca-ba8bf39e1fd8 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.560462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 658/1344] 10.222.0.39 () {54 vars in 912 bytes} [Mon Oct 24 08:50:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 24 08:50:53.566413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c315d-756a-4360-a71b-8933f11d6def None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.566692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c315d-756a-4360-a71b-8933f11d6def None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.566968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c315d-756a-4360-a71b-8933f11d6def None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.581283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a86ea-8e25-4007-a84c-6866e39d662f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:53.620888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5c7ef3-f298-47aa-a721-00ca057544c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:53.622531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5c7ef3-f298-47aa-a721-00ca057544c4 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.622841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5c7ef3-f298-47aa-a721-00ca057544c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.623261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5c7ef3-f298-47aa-a721-00ca057544c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.624480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5c7ef3-f298-47aa-a721-00ca057544c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:53.627033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5c7ef3-f298-47aa-a721-00ca057544c4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:53.633023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-ea4c315d-756a-4360-a71b-8933f11d6def None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:53.635258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ea4c315d-756a-4360-a71b-8933f11d6def None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:53.651581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 687/1345] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:53 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:50:53.655579 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5c7ef3-f298-47aa-a721-00ca057544c4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:53.660956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 659/1346] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:50:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:53.682289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4c315d-756a-4360-a71b-8933f11d6def None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:53.684267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ea4c315d-756a-4360-a71b-8933f11d6def None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:53.708655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5c7ef3-f298-47aa-a721-00ca057544c4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:53.722407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574c2ea9-ac1b-4f31-8bc5-660635fef930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:53.726928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574c2ea9-ac1b-4f31-8bc5-660635fef930 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.727293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574c2ea9-ac1b-4f31-8bc5-660635fef930 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.727580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574c2ea9-ac1b-4f31-8bc5-660635fef930 admin admin] PATH_INFO: `/v3/projects/121ea9da698f48179501653720bb5b98/users/39ecb7ce149f401babf1a6ef3c9383e2/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.770099 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574c2ea9-ac1b-4f31-8bc5-660635fef930 admin admin] RBAC: Authorizing `identity:create_grant(project_id=121ea9da698f48179501653720bb5b98, user_id=39ecb7ce149f401babf1a6ef3c9383e2, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:53.774288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:53.781978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574c2ea9-ac1b-4f31-8bc5-660635fef930 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:53.807669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4c315d-756a-4360-a71b-8933f11d6def None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:53.817694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 688/1347] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:53 2022] PUT /identity/v3/projects/121ea9da698f48179501653720bb5b98/users/39ecb7ce149f401babf1a6ef3c9383e2/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:53.822260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574c2ea9-ac1b-4f31-8bc5-660635fef930 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:53.911683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:53.955199 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb749d22-ad4f-4c0f-8130-745431eee6d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:53.956706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb749d22-ad4f-4c0f-8130-745431eee6d0 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:53.957128 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb749d22-ad4f-4c0f-8130-745431eee6d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:53.957394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb749d22-ad4f-4c0f-8130-745431eee6d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:53.990548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574c2ea9-ac1b-4f31-8bc5-660635fef930 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:54.000639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42067dec-14fb-4d6d-bed0-ffed9c7f531f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:54.002018 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42067dec-14fb-4d6d-bed0-ffed9c7f531f admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:54.002188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42067dec-14fb-4d6d-bed0-ffed9c7f531f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:54.006757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42067dec-14fb-4d6d-bed0-ffed9c7f531f admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:54.010272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42067dec-14fb-4d6d-bed0-ffed9c7f531f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:54.016324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42067dec-14fb-4d6d-bed0-ffed9c7f531f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:54.047470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 689/1348] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:50:53 2022] GET /identity/v3/roles => generated 1420 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:50:54.053016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42067dec-14fb-4d6d-bed0-ffed9c7f531f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:54.072270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42067dec-14fb-4d6d-bed0-ffed9c7f531f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:54.086549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a6c014-3e4f-4d17-9c00-f15403a18d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:54.088049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a6c014-3e4f-4d17-9c00-f15403a18d14 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:54.088049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a6c014-3e4f-4d17-9c00-f15403a18d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:54.088049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a6c014-3e4f-4d17-9c00-f15403a18d14 admin admin] PATH_INFO: `/v3/projects/121ea9da698f48179501653720bb5b98/users/39ecb7ce149f401babf1a6ef3c9383e2/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:54.089794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a6c014-3e4f-4d17-9c00-f15403a18d14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=121ea9da698f48179501653720bb5b98, user_id=39ecb7ce149f401babf1a6ef3c9383e2, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:54.092808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a6c014-3e4f-4d17-9c00-f15403a18d14 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:54.120996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 690/1349] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:50:54 2022] PUT /identity/v3/projects/121ea9da698f48179501653720bb5b98/users/39ecb7ce149f401babf1a6ef3c9383e2/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:50:54.135199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af78859e-10b1-4267-9ad6-ff45cd04d1d7 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:54.135540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af78859e-10b1-4267-9ad6-ff45cd04d1d7 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:54.135822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af78859e-10b1-4267-9ad6-ff45cd04d1d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:54.161426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb749d22-ad4f-4c0f-8130-745431eee6d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:54.162925 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb749d22-ad4f-4c0f-8130-745431eee6d0 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:54.202701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 660/1350] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:54.215114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-af78859e-10b1-4267-9ad6-ff45cd04d1d7 None None] MFA Rules not processed for user `39ecb7ce149f401babf1a6ef3c9383e2`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:54.220871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-af78859e-10b1-4267-9ad6-ff45cd04d1d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:54.480281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 691/1351] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:54 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 346 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:54.597985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb749d22-ad4f-4c0f-8130-745431eee6d0 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:54.608271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb749d22-ad4f-4c0f-8130-745431eee6d0 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:54.793666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807b2ef4-60f2-4baf-9716-9f37611afc36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:54.794297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b2ef4-60f2-4baf-9716-9f37611afc36 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:54.794693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b2ef4-60f2-4baf-9716-9f37611afc36 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:54.794986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b2ef4-60f2-4baf-9716-9f37611afc36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:54.828817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807b2ef4-60f2-4baf-9716-9f37611afc36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:54.831973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807b2ef4-60f2-4baf-9716-9f37611afc36 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:54.860815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 661/1352] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:54 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:54.945590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af78859e-10b1-4267-9ad6-ff45cd04d1d7 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:54.958390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af78859e-10b1-4267-9ad6-ff45cd04d1d7 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:54.963300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d594b1f-1b9e-4645-95fe-550105d02d4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:54.964614 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d594b1f-1b9e-4645-95fe-550105d02d4a service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:54.964952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d594b1f-1b9e-4645-95fe-550105d02d4a service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:54.965290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d594b1f-1b9e-4645-95fe-550105d02d4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:55.134737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d594b1f-1b9e-4645-95fe-550105d02d4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:55.136640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d594b1f-1b9e-4645-95fe-550105d02d4a service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:55.142655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf3456b-0343-4d16-b421-84e751580dbc None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:55.142945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf3456b-0343-4d16-b421-84e751580dbc None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:55.143275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf3456b-0343-4d16-b421-84e751580dbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:55.150661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 692/1353] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:54 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:55.205996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2cf3456b-0343-4d16-b421-84e751580dbc None None] MFA Rules not processed for user `7127613bde784330bec364f02180291e`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:55.207461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2cf3456b-0343-4d16-b421-84e751580dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:55.299641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:55.363558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 662/1354] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:55 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:55.373915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c282e-49a7-4fc0-abe9-1af71aceb4e4 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:55.374605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c282e-49a7-4fc0-abe9-1af71aceb4e4 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:55.375093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c282e-49a7-4fc0-abe9-1af71aceb4e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:55.406611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-874c282e-49a7-4fc0-abe9-1af71aceb4e4 None None] MFA Rules not processed for user `7127613bde784330bec364f02180291e`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:55.409013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-874c282e-49a7-4fc0-abe9-1af71aceb4e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:55.432195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 693/1355] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:55 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:55.445757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf3456b-0343-4d16-b421-84e751580dbc None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:55.448469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2cf3456b-0343-4d16-b421-84e751580dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:55.705950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf3456b-0343-4d16-b421-84e751580dbc None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:55.733900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3946c035-e8a6-45f1-8ffd-b2ca2d07feb1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:55.734530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3946c035-e8a6-45f1-8ffd-b2ca2d07feb1 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:55.734806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3946c035-e8a6-45f1-8ffd-b2ca2d07feb1 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:55.735074 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3946c035-e8a6-45f1-8ffd-b2ca2d07feb1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:55.757167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3946c035-e8a6-45f1-8ffd-b2ca2d07feb1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:55.758775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3946c035-e8a6-45f1-8ffd-b2ca2d07feb1 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:55.776521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 663/1356] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:55 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:56.577058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f535a998-5914-4d5e-85f7-d6056afde91d None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:56.578598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f535a998-5914-4d5e-85f7-d6056afde91d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:56.579178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f535a998-5914-4d5e-85f7-d6056afde91d None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:56.582015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 694/1357] 10.222.0.39 () {54 vars in 912 bytes} [Mon Oct 24 08:50:56 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:50:56.587515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371328bf-f42a-4a0f-99c7-2ea26281b27e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:56.588066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371328bf-f42a-4a0f-99c7-2ea26281b27e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:56.588682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371328bf-f42a-4a0f-99c7-2ea26281b27e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:56.643836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-371328bf-f42a-4a0f-99c7-2ea26281b27e None None] MFA Rules not processed for user `8bfae94e0fa041c9802729370a810245`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:56.645766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-371328bf-f42a-4a0f-99c7-2ea26281b27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:56.777999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 664/1358] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:50:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:50:56.798157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f535a998-5914-4d5e-85f7-d6056afde91d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:56.803203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f535a998-5914-4d5e-85f7-d6056afde91d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:56.904092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f535a998-5914-4d5e-85f7-d6056afde91d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:56.938564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6db319-9def-4bdc-bb6a-216165cc1902 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:56.940126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6db319-9def-4bdc-bb6a-216165cc1902 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:56.940504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6db319-9def-4bdc-bb6a-216165cc1902 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:56.940777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6db319-9def-4bdc-bb6a-216165cc1902 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:56.978885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6db319-9def-4bdc-bb6a-216165cc1902 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:56.980634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6db319-9def-4bdc-bb6a-216165cc1902 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:57.001853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 695/1359] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:50:57.887056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371328bf-f42a-4a0f-99c7-2ea26281b27e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:50:57.889818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-371328bf-f42a-4a0f-99c7-2ea26281b27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:58.090165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df710a3d-d671-400e-b1f3-a9923a499767 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:58.091038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df710a3d-d671-400e-b1f3-a9923a499767 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:58.091471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df710a3d-d671-400e-b1f3-a9923a499767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:58.132705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:50:58.136053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-df710a3d-d671-400e-b1f3-a9923a499767 None None] MFA Rules not processed for user `7127613bde784330bec364f02180291e`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:50:58.137499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-df710a3d-d671-400e-b1f3-a9923a499767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:58.162588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 696/1360] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:50:58 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:50:58.177702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371328bf-f42a-4a0f-99c7-2ea26281b27e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:50:58.195736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda0682c-9a51-4801-a3b0-890a659b5f92 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:50:58.196407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda0682c-9a51-4801-a3b0-890a659b5f92 service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:50:58.196694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda0682c-9a51-4801-a3b0-890a659b5f92 service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:50:58.196974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda0682c-9a51-4801-a3b0-890a659b5f92 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:50:58.206208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bda0682c-9a51-4801-a3b0-890a659b5f92 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:50:58.229684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda0682c-9a51-4801-a3b0-890a659b5f92 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:50:58.231320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda0682c-9a51-4801-a3b0-890a659b5f92 service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:50:58.246449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 665/1361] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:50:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:02.054256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27889d04-406f-40bb-9119-1b84f6faa055 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:02.055265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27889d04-406f-40bb-9119-1b84f6faa055 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:02.056315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27889d04-406f-40bb-9119-1b84f6faa055 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:02.090110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:02.114513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-27889d04-406f-40bb-9119-1b84f6faa055 None None] MFA Rules not processed for user `701e4507214f4634ad09e32e21749a35`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:02.122121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-27889d04-406f-40bb-9119-1b84f6faa055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:02.376175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 697/1362] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:02.391126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980a03c0-da86-4940-9384-5e6ed6b70c77 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:02.393105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980a03c0-da86-4940-9384-5e6ed6b70c77 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:02.393546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980a03c0-da86-4940-9384-5e6ed6b70c77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:02.459649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-980a03c0-da86-4940-9384-5e6ed6b70c77 None None] MFA Rules not processed for user `701e4507214f4634ad09e32e21749a35`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:02.461437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-980a03c0-da86-4940-9384-5e6ed6b70c77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:02.478130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 666/1363] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:02.731809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27889d04-406f-40bb-9119-1b84f6faa055 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:02.737175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb49d96-0d7c-4ae8-b8d0-eeafd101c197 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:02.737464 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb49d96-0d7c-4ae8-b8d0-eeafd101c197 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:02.737748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb49d96-0d7c-4ae8-b8d0-eeafd101c197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:02.794382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27889d04-406f-40bb-9119-1b84f6faa055 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:02.819679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524d9969-8cb4-4b0a-870b-e839e700f30b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:02.821916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d9969-8cb4-4b0a-870b-e839e700f30b service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:02.826036 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d9969-8cb4-4b0a-870b-e839e700f30b service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:02.827574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d9969-8cb4-4b0a-870b-e839e700f30b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:02.864414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524d9969-8cb4-4b0a-870b-e839e700f30b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:02.867667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524d9969-8cb4-4b0a-870b-e839e700f30b service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:02.904061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 698/1364] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:02 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:02.914547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524d9969-8cb4-4b0a-870b-e839e700f30b service nova] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:02.922647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5eb49d96-0d7c-4ae8-b8d0-eeafd101c197 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:02.934947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5eb49d96-0d7c-4ae8-b8d0-eeafd101c197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:03.172694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524d9969-8cb4-4b0a-870b-e839e700f30b service nova] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:03.179864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:03.180440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:03.180722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:03.180989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] PATH_INFO: `/v3/users/3bc80d8fab5a4f9bbdc605a7689f21fe` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:03.184205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bc80d8fab5a4f9bbdc605a7689f21fe)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:03.186440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:03.236520 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 667/1365] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:51:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 501 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:03.324407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bc80d8fab5a4f9bbdc605a7689f21fe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:03.350517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bc80d8fab5a4f9bbdc605a7689f21fe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:03.399512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bc80d8fab5a4f9bbdc605a7689f21fe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:03.405512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bc80d8fab5a4f9bbdc605a7689f21fe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:03.406602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bc80d8fab5a4f9bbdc605a7689f21fe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:03.463318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 699/1366] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:02 2022] DELETE /identity/v3/users/3bc80d8fab5a4f9bbdc605a7689f21fe => generated 0 bytes in 559 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:03.467911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c79cee-cfa7-4200-96af-56798214cfbd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:03.468887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c79cee-cfa7-4200-96af-56798214cfbd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:03.469313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c79cee-cfa7-4200-96af-56798214cfbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:03.523311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c4c79cee-cfa7-4200-96af-56798214cfbd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:03.524583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c4c79cee-cfa7-4200-96af-56798214cfbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:03.558342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:03.560157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:03.606164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:03.772574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 668/1367] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:04.031565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99599973-1193-4e14-b796-44a8d6f9a8fe admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:04.066665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:04.067314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:04.067603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:04.067875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:04.080098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:04.335214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e58410-051d-430a-bf23-8e0186881e3e None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:04.335660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e58410-051d-430a-bf23-8e0186881e3e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:04.336067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e58410-051d-430a-bf23-8e0186881e3e None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:04.337659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 669/1368] 10.222.0.39 () {54 vars in 912 bytes} [Mon Oct 24 08:51:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:51:04.346278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8317bc1-19a1-462c-9bfc-10d3d4aed4e0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:04.346887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8317bc1-19a1-462c-9bfc-10d3d4aed4e0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:04.347295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8317bc1-19a1-462c-9bfc-10d3d4aed4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:04.375980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:04.378311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:04.389066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 700/1369] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:51:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 831 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 24 08:51:04.390316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:04.391492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:04.410564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b8317bc1-19a1-462c-9bfc-10d3d4aed4e0 None None] MFA Rules not processed for user `8bfae94e0fa041c9802729370a810245`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:04.413849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b8317bc1-19a1-462c-9bfc-10d3d4aed4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:04.507839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2289e540-1d55-4bb9-b074-07668808d3c0 service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:04.516148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:04.517400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:04.517798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:04.518018 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:04.530142 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:04.531758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:04.542648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:04.569893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 701/1370] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:04.588827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 670/1371] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:51:04 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:04.870316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:04.871365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:04.902845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a94cd8e-b12f-4b49-b2d6-103bb61c7f12 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:04.936697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba426a6-43b4-4877-9543-698b26c1a066 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:04.944786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba426a6-43b4-4877-9543-698b26c1a066 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:04.948636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba426a6-43b4-4877-9543-698b26c1a066 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:04.955801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba426a6-43b4-4877-9543-698b26c1a066 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:05.021850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba426a6-43b4-4877-9543-698b26c1a066 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:05.026168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba426a6-43b4-4877-9543-698b26c1a066 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:05.081694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 702/1372] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:05.362510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793a2c3-91de-49f6-98c1-0a2301f7a021 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:05.362836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793a2c3-91de-49f6-98c1-0a2301f7a021 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:05.363049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793a2c3-91de-49f6-98c1-0a2301f7a021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:05.431621 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b793a2c3-91de-49f6-98c1-0a2301f7a021 None None] MFA Rules not processed for user `8bfae94e0fa041c9802729370a810245`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:05.439531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b793a2c3-91de-49f6-98c1-0a2301f7a021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:05.480174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 671/1373] 10.222.0.13 () {58 vars in 1015 bytes} [Mon Oct 24 08:51:05 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:05.525317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba426a6-43b4-4877-9543-698b26c1a066 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:05.538124 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba426a6-43b4-4877-9543-698b26c1a066 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:05.558132 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:05.558687 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:05.558971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:05.559248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:05.591304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:05.592841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:05.643705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 703/1374] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:07.708631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8e7c65-6e77-4c70-980f-b01468f769b0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:07.710130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8e7c65-6e77-4c70-980f-b01468f769b0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:07.711428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8e7c65-6e77-4c70-980f-b01468f769b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:07.803187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3b8e7c65-6e77-4c70-980f-b01468f769b0 None None] MFA Rules not processed for user `552c23b8456a40ec8e01297aca7965d3`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:07.805801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3b8e7c65-6e77-4c70-980f-b01468f769b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:07.824907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:07.827357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:07.882295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38928b19-e3e1-4a0b-af2d-6fd48edebda8 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:07.902404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:07.903111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:07.903381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:07.903657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] PATH_INFO: `/v3/projects/e4d18888ab7a4325b341d6a2a63ef7d6` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:07.915014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4d18888ab7a4325b341d6a2a63ef7d6)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:07.951039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:07.959141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:08.076536 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 672/1375] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:07 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 369 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:08.095247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8e7c65-6e77-4c70-980f-b01468f769b0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:08.098025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3b8e7c65-6e77-4c70-980f-b01468f769b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:08.296005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4d18888ab7a4325b341d6a2a63ef7d6', 'request_id': 'req-cada4cb0-6d07-466b-9e28-407774aa2cea'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.332620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4d18888ab7a4325b341d6a2a63ef7d6', 'request_id': 'req-cada4cb0-6d07-466b-9e28-407774aa2cea'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.354879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4d18888ab7a4325b341d6a2a63ef7d6', 'request_id': 'req-cada4cb0-6d07-466b-9e28-407774aa2cea'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.412454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] The token cache is being invalidate because project e4d18888ab7a4325b341d6a2a63ef7d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:08.414988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.416387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.432009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 704/1376] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:07 2022] DELETE /identity/v3/projects/e4d18888ab7a4325b341d6a2a63ef7d6 => generated 0 bytes in 602 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:08.433126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:08.434665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:08.473342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8e7c65-6e77-4c70-980f-b01468f769b0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:08.581970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cada4cb0-6d07-466b-9e28-407774aa2cea admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:08.607524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:08.609571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:08.611459 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:08.614988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] PATH_INFO: `/v3/users/39ecb7ce149f401babf1a6ef3c9383e2` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:08.617880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39ecb7ce149f401babf1a6ef3c9383e2)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:08.623827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:08.693838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a9957c-adfe-48fd-a09a-154975bcc0a5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:08.694535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9957c-adfe-48fd-a09a-154975bcc0a5 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:08.694816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9957c-adfe-48fd-a09a-154975bcc0a5 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:08.695089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9957c-adfe-48fd-a09a-154975bcc0a5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:08.705604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-60a9957c-adfe-48fd-a09a-154975bcc0a5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:08.767147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:08.830468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39ecb7ce149f401babf1a6ef3c9383e2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.856480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39ecb7ce149f401babf1a6ef3c9383e2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.901843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39ecb7ce149f401babf1a6ef3c9383e2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.903640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39ecb7ce149f401babf1a6ef3c9383e2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.908813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39ecb7ce149f401babf1a6ef3c9383e2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:08.935123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 705/1377] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:08 2022] DELETE /identity/v3/users/39ecb7ce149f401babf1a6ef3c9383e2 => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:08.946869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a9957c-adfe-48fd-a09a-154975bcc0a5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:08.949640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a9957c-adfe-48fd-a09a-154975bcc0a5 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:09.001568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 673/1378] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:08 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 907 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:09.618009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:09.619704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:09.667483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:09.879795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbcd1b8-a2b6-42e2-b7a9-e07cb5959c35 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:09.887556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:09.890918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:09.892092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:09.892550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] PATH_INFO: `/v3/users/6661da2c7e754749a15bf615c23fb933` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:09.895002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6661da2c7e754749a15bf615c23fb933)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:09.897815 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:10.078552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6661da2c7e754749a15bf615c23fb933'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:10.123647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6661da2c7e754749a15bf615c23fb933'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:10.193663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6661da2c7e754749a15bf615c23fb933'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:10.196743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6661da2c7e754749a15bf615c23fb933'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:10.200228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6661da2c7e754749a15bf615c23fb933'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:10.273529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 706/1379] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:09 2022] DELETE /identity/v3/users/6661da2c7e754749a15bf615c23fb933 => generated 0 bytes in 656 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:10.277803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa684b4-fcd0-4d3b-aa60-89f4bf913f54 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:10.278286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa684b4-fcd0-4d3b-aa60-89f4bf913f54 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:10.278414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa684b4-fcd0-4d3b-aa60-89f4bf913f54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:10.376604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-0aa684b4-fcd0-4d3b-aa60-89f4bf913f54 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:10.383403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0aa684b4-fcd0-4d3b-aa60-89f4bf913f54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:10.728370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 674/1380] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 452 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:10.803732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:10.805431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:10.927808 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc62159-7690-44b0-962e-eb4b6e826168 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:10.928104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc62159-7690-44b0-962e-eb4b6e826168 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:10.928408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc62159-7690-44b0-962e-eb4b6e826168 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:10.930417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 675/1381] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:51:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:51:11.037465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c12630-68c1-4ba0-8032-820d6a38ffe3 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:11.048953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8270e371-d48b-4282-a7b0-cb22b77086f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:11.051134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8270e371-d48b-4282-a7b0-cb22b77086f5 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:11.051537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8270e371-d48b-4282-a7b0-cb22b77086f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:11.051930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8270e371-d48b-4282-a7b0-cb22b77086f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:11.076981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8270e371-d48b-4282-a7b0-cb22b77086f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:11.078590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8270e371-d48b-4282-a7b0-cb22b77086f5 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:11.110699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 707/1382] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:11.627309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa87a84a-b9bf-49fc-90c2-cdc84edbd008 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:11.627807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa87a84a-b9bf-49fc-90c2-cdc84edbd008 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:11.627884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa87a84a-b9bf-49fc-90c2-cdc84edbd008 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:11.699095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-aa87a84a-b9bf-49fc-90c2-cdc84edbd008 None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:11.702843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-aa87a84a-b9bf-49fc-90c2-cdc84edbd008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:11.714973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3c0e0-dea8-4571-958a-39977169a982 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:11.715651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3c0e0-dea8-4571-958a-39977169a982 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:11.716285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3c0e0-dea8-4571-958a-39977169a982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:11.820434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-83a3c0e0-dea8-4571-958a-39977169a982 None None] MFA Rules not processed for user `ebdef1d69bb14837b9a0c28cbba332fc`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:11.825006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-83a3c0e0-dea8-4571-958a-39977169a982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:11.894352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:11.982124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 676/1383] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:51:11 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 355 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:12.086125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 708/1384] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:11 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 372 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:12.101065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa87a84a-b9bf-49fc-90c2-cdc84edbd008 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:12.102752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-aa87a84a-b9bf-49fc-90c2-cdc84edbd008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:12.124916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa87a84a-b9bf-49fc-90c2-cdc84edbd008 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:12.142366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:12.143279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:12.143702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:12.144123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:12.164790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:12.167286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:12.195130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 677/1385] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:12 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:12.220636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a3c0e0-dea8-4571-958a-39977169a982 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:12.225587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-83a3c0e0-dea8-4571-958a-39977169a982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:12.287009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a3c0e0-dea8-4571-958a-39977169a982 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:12.318253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ff6361-400f-4dc0-8411-7aa746ca32a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:12.319476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ff6361-400f-4dc0-8411-7aa746ca32a2 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:12.319923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ff6361-400f-4dc0-8411-7aa746ca32a2 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:12.320534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ff6361-400f-4dc0-8411-7aa746ca32a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:12.339517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ff6361-400f-4dc0-8411-7aa746ca32a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:12.345997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ff6361-400f-4dc0-8411-7aa746ca32a2 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:12.395767 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 709/1386] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:13.450923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:13.456231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:13.515715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe670adf-09d3-47f0-bd03-c35e02347f41 service nova] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:13.543546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:13.547826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:13.550325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:13.551982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] PATH_INFO: `/v3/projects/2b2bf942072c4ef684069f485877d379` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:13.561623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b2bf942072c4ef684069f485877d379)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:13.581453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:13.848540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:13.901697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b2bf942072c4ef684069f485877d379', 'request_id': 'req-64568b68-0b28-41bd-9cdd-616c6e347194'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:13.925940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b2bf942072c4ef684069f485877d379', 'request_id': 'req-64568b68-0b28-41bd-9cdd-616c6e347194'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:13.944985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b2bf942072c4ef684069f485877d379', 'request_id': 'req-64568b68-0b28-41bd-9cdd-616c6e347194'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:13.991399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] The token cache is being invalidate because project 2b2bf942072c4ef684069f485877d379 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:13.992272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:13.998935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64568b68-0b28-41bd-9cdd-616c6e347194 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:14.014649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 678/1387] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:13 2022] DELETE /identity/v3/projects/2b2bf942072c4ef684069f485877d379 => generated 0 bytes in 565 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:14.023051 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6781647a-cb59-4418-be07-b8a5c3ae584c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:14.023580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6781647a-cb59-4418-be07-b8a5c3ae584c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:14.023580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6781647a-cb59-4418-be07-b8a5c3ae584c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:14.074487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6781647a-cb59-4418-be07-b8a5c3ae584c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:14.079830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6781647a-cb59-4418-be07-b8a5c3ae584c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:14.311792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 710/1388] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:14.312149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6affd6-1021-4b1c-a858-13866758f3ec None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:14.312383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6affd6-1021-4b1c-a858-13866758f3ec None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:14.312671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6affd6-1021-4b1c-a858-13866758f3ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:14.385892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5f6affd6-1021-4b1c-a858-13866758f3ec None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:14.386182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5f6affd6-1021-4b1c-a858-13866758f3ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:14.421120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 679/1389] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:14.430525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6781647a-cb59-4418-be07-b8a5c3ae584c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:14.462855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6781647a-cb59-4418-be07-b8a5c3ae584c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:14.480520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:14.481332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:14.482003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:14.483443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:14.484161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:14.484482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:14.489554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:14.532813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 711/1390] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:14.534497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6affd6-1021-4b1c-a858-13866758f3ec None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:14.581738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6affd6-1021-4b1c-a858-13866758f3ec None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:14.628955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa2255b-cf18-40e4-a2a5-8cf0b3e7ebb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:14.629606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa2255b-cf18-40e4-a2a5-8cf0b3e7ebb0 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:14.629884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa2255b-cf18-40e4-a2a5-8cf0b3e7ebb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:14.630153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa2255b-cf18-40e4-a2a5-8cf0b3e7ebb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:14.630831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa2255b-cf18-40e4-a2a5-8cf0b3e7ebb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:14.632958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa2255b-cf18-40e4-a2a5-8cf0b3e7ebb0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:14.734595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 680/1391] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:14 2022] POST /identity/v3/projects => generated 373 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:14.735525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:14.783715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58633f6-f953-4017-aad6-c5cdbef60ff7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:14.992745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255f459e-c4b7-4c34-ba8d-e8291b16cc24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:14.993431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255f459e-c4b7-4c34-ba8d-e8291b16cc24 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:14.994046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255f459e-c4b7-4c34-ba8d-e8291b16cc24 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:14.994046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255f459e-c4b7-4c34-ba8d-e8291b16cc24 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:14.997975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255f459e-c4b7-4c34-ba8d-e8291b16cc24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:15.000118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255f459e-c4b7-4c34-ba8d-e8291b16cc24 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:15.067035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa2255b-cf18-40e4-a2a5-8cf0b3e7ebb0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:15.077428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 712/1392] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:14 2022] POST /identity/v3/users => generated 287 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:15.083103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa2255b-cf18-40e4-a2a5-8cf0b3e7ebb0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:15.091745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648dbd15-0ed2-434b-b93b-938cb55261ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:15.092787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648dbd15-0ed2-434b-b93b-938cb55261ff admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:15.093343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648dbd15-0ed2-434b-b93b-938cb55261ff admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:15.094043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648dbd15-0ed2-434b-b93b-938cb55261ff admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:15.095892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648dbd15-0ed2-434b-b93b-938cb55261ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:15.098095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648dbd15-0ed2-434b-b93b-938cb55261ff admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:15.127029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 681/1393] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:15 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:15.127612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255f459e-c4b7-4c34-ba8d-e8291b16cc24 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:15.147544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255f459e-c4b7-4c34-ba8d-e8291b16cc24 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:15.193034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857c496d-aedc-4af8-a70f-a93ae00d0bae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:15.194655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c496d-aedc-4af8-a70f-a93ae00d0bae admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:15.195192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c496d-aedc-4af8-a70f-a93ae00d0bae admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:15.195915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c496d-aedc-4af8-a70f-a93ae00d0bae admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:15.197163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857c496d-aedc-4af8-a70f-a93ae00d0bae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:15.199356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857c496d-aedc-4af8-a70f-a93ae00d0bae admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:15.222518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 713/1394] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:15 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:15.228742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648dbd15-0ed2-434b-b93b-938cb55261ff admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:15.254123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648dbd15-0ed2-434b-b93b-938cb55261ff admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:15.265747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:15.267652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:15.268157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:15.269577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] PATH_INFO: `/v3/projects/94221c3e732d4b6994f330e0320dfb05/users/b414191fa13349dc93babf241c4095d2/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:15.314127 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] RBAC: Authorizing `identity:create_grant(project_id=94221c3e732d4b6994f330e0320dfb05, user_id=b414191fa13349dc93babf241c4095d2, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:15.322438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:15.372441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 682/1395] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:15 2022] PUT /identity/v3/projects/94221c3e732d4b6994f330e0320dfb05/users/b414191fa13349dc93babf241c4095d2/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:15.389479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622805a2-8257-4edf-bcef-87fd247b2ed8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:15.389760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622805a2-8257-4edf-bcef-87fd247b2ed8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:15.390601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622805a2-8257-4edf-bcef-87fd247b2ed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:15.455042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:15.459491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-622805a2-8257-4edf-bcef-87fd247b2ed8 None None] MFA Rules not processed for user `b414191fa13349dc93babf241c4095d2`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:15.464232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-622805a2-8257-4edf-bcef-87fd247b2ed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:15.675696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 714/1396] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:15.690071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:15.692690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:15.947388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ffe815-1e9d-4ff0-98f2-9fb9261919db admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:15.965194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:15.966472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:15.967097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:15.967878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:16.020267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:16.028479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:16.070522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fe863c-5f87-4afe-8b3b-b7fb9e4d710e None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:16.070814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fe863c-5f87-4afe-8b3b-b7fb9e4d710e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:16.071364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fe863c-5f87-4afe-8b3b-b7fb9e4d710e None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:16.074002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 715/1397] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:51:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:51:16.083235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 683/1398] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:16.137403 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8806f7-574e-4ac7-beb8-55cec5f6e13c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:16.138419 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8806f7-574e-4ac7-beb8-55cec5f6e13c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:16.138836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8806f7-574e-4ac7-beb8-55cec5f6e13c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:16.181250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-3a8806f7-574e-4ac7-beb8-55cec5f6e13c None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:16.186062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3a8806f7-574e-4ac7-beb8-55cec5f6e13c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:16.221805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 716/1399] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:51:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:16.261150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:16.262748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:16.390863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8806f7-574e-4ac7-beb8-55cec5f6e13c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:16.402605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8806f7-574e-4ac7-beb8-55cec5f6e13c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:16.425465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:16.446025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5cd6ba-a93e-4f91-a0a9-88578debc959 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:16.446656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5cd6ba-a93e-4f91-a0a9-88578debc959 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:16.446935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5cd6ba-a93e-4f91-a0a9-88578debc959 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:16.447205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5cd6ba-a93e-4f91-a0a9-88578debc959 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:16.517573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5cd6ba-a93e-4f91-a0a9-88578debc959 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:16.529304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5cd6ba-a93e-4f91-a0a9-88578debc959 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:16.572418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b24380-3a4d-44e7-ba8d-a9c5df797424 service nova] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:16.588483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 717/1400] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:16.600965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:16.601637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:16.604418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:16.604741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] PATH_INFO: `/v3/projects/121ea9da698f48179501653720bb5b98` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:16.607601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=121ea9da698f48179501653720bb5b98)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:16.612893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:16.811206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '121ea9da698f48179501653720bb5b98', 'request_id': 'req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:16.833420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '121ea9da698f48179501653720bb5b98', 'request_id': 'req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:16.849853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '121ea9da698f48179501653720bb5b98', 'request_id': 'req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:16.879453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] The token cache is being invalidate because project 121ea9da698f48179501653720bb5b98 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:16.879753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:16.885026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5dca02cc-db7d-44c3-93ba-22ecbee78bd1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:16.894495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 684/1401] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:16 2022] DELETE /identity/v3/projects/121ea9da698f48179501653720bb5b98 => generated 0 bytes in 630 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:16.904915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64809f-527b-4853-8a5c-59e1b88ddbec None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:16.905264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64809f-527b-4853-8a5c-59e1b88ddbec None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:16.906113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64809f-527b-4853-8a5c-59e1b88ddbec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:17.010847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-2c64809f-527b-4853-8a5c-59e1b88ddbec None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:17.012734 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2c64809f-527b-4853-8a5c-59e1b88ddbec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:17.185446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 718/1402] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:17.193713 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5158676-a517-47ce-b558-153fa3adb887 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:17.194000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5158676-a517-47ce-b558-153fa3adb887 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:17.194282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5158676-a517-47ce-b558-153fa3adb887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:17.226405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e5158676-a517-47ce-b558-153fa3adb887 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:17.245120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e5158676-a517-47ce-b558-153fa3adb887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:17.267065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 685/1403] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:17.271813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c64809f-527b-4853-8a5c-59e1b88ddbec None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:17.288647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c64809f-527b-4853-8a5c-59e1b88ddbec None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:17.312217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:17.315548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:17.316212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:17.316928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:17.318068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:17.318738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:17.322595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:17.337676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5158676-a517-47ce-b558-153fa3adb887 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:17.339746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e5158676-a517-47ce-b558-153fa3adb887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:17.364560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5158676-a517-47ce-b558-153fa3adb887 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:17.377300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:17.377956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:17.378238 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:17.378513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] PATH_INFO: `/v3/users/b414191fa13349dc93babf241c4095d2` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:17.379616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 719/1404] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:17.387241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:17.389572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b414191fa13349dc93babf241c4095d2)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:17.398909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:17.421274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3064fc-78e6-4e8d-9b1b-5c85c47419fa admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:17.456299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b88cdd-f9a6-4eb0-8bb3-9eb9b9d40381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:17.459692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b88cdd-f9a6-4eb0-8bb3-9eb9b9d40381 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:17.459692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b88cdd-f9a6-4eb0-8bb3-9eb9b9d40381 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:17.459692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b88cdd-f9a6-4eb0-8bb3-9eb9b9d40381 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:17.459692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b88cdd-f9a6-4eb0-8bb3-9eb9b9d40381 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:17.461169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b88cdd-f9a6-4eb0-8bb3-9eb9b9d40381 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:17.579052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 720/1405] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:17 2022] POST /identity/v3/projects => generated 387 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:17.586083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b88cdd-f9a6-4eb0-8bb3-9eb9b9d40381 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:17.588015 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b414191fa13349dc93babf241c4095d2'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:17.629548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b414191fa13349dc93babf241c4095d2'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:17.699143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b414191fa13349dc93babf241c4095d2'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:17.705213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b414191fa13349dc93babf241c4095d2'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:17.707568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f89a8c7b-570a-4b3d-81df-b25b2f067b30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b414191fa13349dc93babf241c4095d2'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:17.779829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 686/1406] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:17 2022] DELETE /identity/v3/users/b414191fa13349dc93babf241c4095d2 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:17.787308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a03db-6365-457e-83c2-ca86940a6fe5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:17.787723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a03db-6365-457e-83c2-ca86940a6fe5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:17.788011 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a03db-6365-457e-83c2-ca86940a6fe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:17.873118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:17.873868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-ff5a03db-6365-457e-83c2-ca86940a6fe5 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:17.876569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ff5a03db-6365-457e-83c2-ca86940a6fe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:17.936410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b88cdd-f9a6-4eb0-8bb3-9eb9b9d40381 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:17.952347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:18.054110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 687/1407] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:18.090704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5a03db-6365-457e-83c2-ca86940a6fe5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:18.092636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ff5a03db-6365-457e-83c2-ca86940a6fe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:18.164905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:18.165468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:18.165829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:18.166106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:18.170593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:18.175940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:18.277194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5a03db-6365-457e-83c2-ca86940a6fe5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:18.288738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24159e1-c4d3-48b2-a672-6fc999866282 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:18.289279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24159e1-c4d3-48b2-a672-6fc999866282 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:18.289561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24159e1-c4d3-48b2-a672-6fc999866282 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:18.289829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24159e1-c4d3-48b2-a672-6fc999866282 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:18.315404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24159e1-c4d3-48b2-a672-6fc999866282 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:18.315570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 721/1408] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:17 2022] POST /identity/v3/users => generated 294 bytes in 729 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:18.316012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24159e1-c4d3-48b2-a672-6fc999866282 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:18.319968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:18.322433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:18.343506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 688/1409] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:18.382692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e5d642-4891-4621-8b24-72825e635210 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:18.412155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2592e9d4-f0a2-4d9c-b27a-43420f04c593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:18.412689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592e9d4-f0a2-4d9c-b27a-43420f04c593 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:18.413024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592e9d4-f0a2-4d9c-b27a-43420f04c593 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:18.413296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592e9d4-f0a2-4d9c-b27a-43420f04c593 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:18.413949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2592e9d4-f0a2-4d9c-b27a-43420f04c593 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:18.415557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2592e9d4-f0a2-4d9c-b27a-43420f04c593 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:18.512322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 722/1410] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:18 2022] GET /identity/v3/roles => generated 1420 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:18.516879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24159e1-c4d3-48b2-a672-6fc999866282 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:18.534213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24159e1-c4d3-48b2-a672-6fc999866282 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:18.548804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1d1789-9bfb-4d04-bf35-da5d474b92a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:18.549614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1d1789-9bfb-4d04-bf35-da5d474b92a7 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:18.550051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1d1789-9bfb-4d04-bf35-da5d474b92a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:18.550522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1d1789-9bfb-4d04-bf35-da5d474b92a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:18.551193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1d1789-9bfb-4d04-bf35-da5d474b92a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:18.555894 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1d1789-9bfb-4d04-bf35-da5d474b92a7 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:18.591915 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 689/1411] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:18 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:18.598304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2592e9d4-f0a2-4d9c-b27a-43420f04c593 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:18.644716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2592e9d4-f0a2-4d9c-b27a-43420f04c593 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:18.671494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab7d48-d352-46cd-ae2c-af374121b20b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:18.673722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab7d48-d352-46cd-ae2c-af374121b20b admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:18.675118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab7d48-d352-46cd-ae2c-af374121b20b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:18.676154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab7d48-d352-46cd-ae2c-af374121b20b admin admin] PATH_INFO: `/v3/projects/942ec1fa400e4f5fbc8cef75e25775da/users/2e6f7b2c82b34c3087ef98efe9164ee8/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:18.760973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ab7d48-d352-46cd-ae2c-af374121b20b admin admin] RBAC: Authorizing `identity:create_grant(project_id=942ec1fa400e4f5fbc8cef75e25775da, user_id=2e6f7b2c82b34c3087ef98efe9164ee8, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:18.760973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ab7d48-d352-46cd-ae2c-af374121b20b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:18.807790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 723/1412] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:18 2022] PUT /identity/v3/projects/942ec1fa400e4f5fbc8cef75e25775da/users/2e6f7b2c82b34c3087ef98efe9164ee8/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:18.810551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae953a8d-beb1-4f68-b9e8-207c0d0555f3 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:18.810551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae953a8d-beb1-4f68-b9e8-207c0d0555f3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:18.810945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae953a8d-beb1-4f68-b9e8-207c0d0555f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:18.883970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-ae953a8d-beb1-4f68-b9e8-207c0d0555f3 None None] MFA Rules not processed for user `2e6f7b2c82b34c3087ef98efe9164ee8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:18.886440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ae953a8d-beb1-4f68-b9e8-207c0d0555f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:19.119592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 690/1413] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:18 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:19.122678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab7d48-d352-46cd-ae2c-af374121b20b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:19.276597 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab7d48-d352-46cd-ae2c-af374121b20b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:19.282184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32c24ef-e1cf-4944-8799-727f78f0de1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:19.283355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c24ef-e1cf-4944-8799-727f78f0de1a admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:19.283839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c24ef-e1cf-4944-8799-727f78f0de1a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:19.284367 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c24ef-e1cf-4944-8799-727f78f0de1a admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:19.286564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32c24ef-e1cf-4944-8799-727f78f0de1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:19.300512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32c24ef-e1cf-4944-8799-727f78f0de1a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:19.354450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 724/1414] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:19 2022] POST /identity/v3/projects => generated 385 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:19.359456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae953a8d-beb1-4f68-b9e8-207c0d0555f3 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:19.397710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae953a8d-beb1-4f68-b9e8-207c0d0555f3 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:19.570397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:19.609488 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c4d7d-7a0c-4409-b11d-657ee0ea83f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:19.610768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c4d7d-7a0c-4409-b11d-657ee0ea83f1 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:19.611075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c4d7d-7a0c-4409-b11d-657ee0ea83f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:19.611535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c4d7d-7a0c-4409-b11d-657ee0ea83f1 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:19.612292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c4d7d-7a0c-4409-b11d-657ee0ea83f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:19.616392 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c4d7d-7a0c-4409-b11d-657ee0ea83f1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:19.695934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 691/1415] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:19 2022] POST /identity/v3/users => generated 299 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:19.697245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32c24ef-e1cf-4944-8799-727f78f0de1a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:19.710409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32c24ef-e1cf-4944-8799-727f78f0de1a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:19.718327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d483e32e-af5a-4b11-b1a2-538f3ecb2580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:19.719748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d483e32e-af5a-4b11-b1a2-538f3ecb2580 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:19.720993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d483e32e-af5a-4b11-b1a2-538f3ecb2580 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:19.721670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d483e32e-af5a-4b11-b1a2-538f3ecb2580 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:19.722799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d483e32e-af5a-4b11-b1a2-538f3ecb2580 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:19.725627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d483e32e-af5a-4b11-b1a2-538f3ecb2580 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:19.747523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 725/1416] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:19 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:19.750856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c4d7d-7a0c-4409-b11d-657ee0ea83f1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:19.767531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c4d7d-7a0c-4409-b11d-657ee0ea83f1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:19.773569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:19.774633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:19.775051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:19.776126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] PATH_INFO: `/v3/projects/01578563aeac46149e0b4abe67303101/users/6810668626a44cdaa4f830cb892acadf/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:19.867486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01578563aeac46149e0b4abe67303101, user_id=6810668626a44cdaa4f830cb892acadf, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:19.875184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:19.913835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 692/1417] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:19 2022] PUT /identity/v3/projects/01578563aeac46149e0b4abe67303101/users/6810668626a44cdaa4f830cb892acadf/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:19.915585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d483e32e-af5a-4b11-b1a2-538f3ecb2580 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:19.942656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d483e32e-af5a-4b11-b1a2-538f3ecb2580 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:20.019260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:20.074741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:20.076856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:20.143886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e5689a-5112-400e-bb93-c556320046d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:20.144822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e5689a-5112-400e-bb93-c556320046d8 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:20.145225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e5689a-5112-400e-bb93-c556320046d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:20.145652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e5689a-5112-400e-bb93-c556320046d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:20.146547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e5689a-5112-400e-bb93-c556320046d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:20.148522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e5689a-5112-400e-bb93-c556320046d8 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:20.172624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 726/1418] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:19 2022] GET /identity/v3/roles => generated 1420 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:20.176853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e5689a-5112-400e-bb93-c556320046d8 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:20.201830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e5689a-5112-400e-bb93-c556320046d8 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:20.214933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003d67f5-6e09-4801-b242-0267795fef22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:20.215758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d67f5-6e09-4801-b242-0267795fef22 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:20.216323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d67f5-6e09-4801-b242-0267795fef22 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:20.216801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d67f5-6e09-4801-b242-0267795fef22 admin admin] PATH_INFO: `/v3/projects/01578563aeac46149e0b4abe67303101/users/6810668626a44cdaa4f830cb892acadf/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:20.221800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003d67f5-6e09-4801-b242-0267795fef22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01578563aeac46149e0b4abe67303101, user_id=6810668626a44cdaa4f830cb892acadf, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:20.241634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003d67f5-6e09-4801-b242-0267795fef22 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:20.278955 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 727/1419] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:20 2022] PUT /identity/v3/projects/01578563aeac46149e0b4abe67303101/users/6810668626a44cdaa4f830cb892acadf/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:20.291293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117dfab-6957-440d-a8e1-ba5d9d981adf None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:20.292215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117dfab-6957-440d-a8e1-ba5d9d981adf None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:20.292925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117dfab-6957-440d-a8e1-ba5d9d981adf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:20.307156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a3b6db-e434-449e-ba00-84fe8fc173c6 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:20.413473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0117dfab-6957-440d-a8e1-ba5d9d981adf None None] MFA Rules not processed for user `6810668626a44cdaa4f830cb892acadf`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:20.420694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0117dfab-6957-440d-a8e1-ba5d9d981adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:20.504835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:20.505508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:20.505789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:20.506076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] PATH_INFO: `/v3/users/0f9689838e2d45039b78ddac79d77df6` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:20.508451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f9689838e2d45039b78ddac79d77df6)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:20.510409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:20.718366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f9689838e2d45039b78ddac79d77df6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:20.766699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f9689838e2d45039b78ddac79d77df6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:20.789357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 728/1420] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:20 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 500 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:20.806119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0117dfab-6957-440d-a8e1-ba5d9d981adf None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:20.807975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0117dfab-6957-440d-a8e1-ba5d9d981adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:20.811949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:20.824524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f9689838e2d45039b78ddac79d77df6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:20.825319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f9689838e2d45039b78ddac79d77df6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:20.826001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-973b4db3-b633-4f1e-bcab-9638e6ca6881 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f9689838e2d45039b78ddac79d77df6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:20.863456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 693/1421] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:20 2022] DELETE /identity/v3/users/0f9689838e2d45039b78ddac79d77df6 => generated 0 bytes in 790 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:20.868655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c240c-8785-4280-bd0e-6023aed07a31 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:20.869297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c240c-8785-4280-bd0e-6023aed07a31 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:20.869741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c240c-8785-4280-bd0e-6023aed07a31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:20.924753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-cb6c240c-8785-4280-bd0e-6023aed07a31 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:20.931401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cb6c240c-8785-4280-bd0e-6023aed07a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:21.000636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0117dfab-6957-440d-a8e1-ba5d9d981adf None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:21.111213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 694/1422] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:21.134734 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:21.157599 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:21.159028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:21.159475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:21.159921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:21.162986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:21.207889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c240c-8785-4280-bd0e-6023aed07a31 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:21.211329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cb6c240c-8785-4280-bd0e-6023aed07a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:21.350122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c240c-8785-4280-bd0e-6023aed07a31 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:21.351125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:21.354726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:21.360726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:21.363406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:21.364803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:21.364803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:21.393451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 729/1423] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:20 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 587 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:21.415342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:21.417586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:21.446391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 695/1424] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:21.469222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:21.471129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:21.490724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa7be5-4161-4178-95b7-06c29ddcf6c0 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:21.496036 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:21.496959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:21.497339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:21.497730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] PATH_INFO: `/v3/users/2e6f7b2c82b34c3087ef98efe9164ee8` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:21.499022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e6f7b2c82b34c3087ef98efe9164ee8)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:21.501636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:21.605710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e6f7b2c82b34c3087ef98efe9164ee8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.617342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e6f7b2c82b34c3087ef98efe9164ee8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.633035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e6f7b2c82b34c3087ef98efe9164ee8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.633428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e6f7b2c82b34c3087ef98efe9164ee8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.634138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d26a9176-0b54-4594-a8ee-f64c6ca8bb06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e6f7b2c82b34c3087ef98efe9164ee8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.653257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 730/1425] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:21 2022] DELETE /identity/v3/users/2e6f7b2c82b34c3087ef98efe9164ee8 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:21.655194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:21.657332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:21.657493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb295e0-ac36-488a-a665-398e504e7f1b None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:21.658602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb295e0-ac36-488a-a665-398e504e7f1b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:21.658602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb295e0-ac36-488a-a665-398e504e7f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:21.680191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7507b507-851c-4142-a351-363c258d9af1 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:21.689759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-3fb295e0-ac36-488a-a665-398e504e7f1b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:21.693480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3fb295e0-ac36-488a-a665-398e504e7f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:21.805197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 731/1426] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:21.807505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:21.808421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:21.808707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:21.809036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] PATH_INFO: `/v3/projects/94221c3e732d4b6994f330e0320dfb05` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:21.810258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] RBAC: Authorizing `identity:delete_project(project_id=94221c3e732d4b6994f330e0320dfb05)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:21.812265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:21.841775 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb295e0-ac36-488a-a665-398e504e7f1b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:21.844616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3fb295e0-ac36-488a-a665-398e504e7f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:21.861363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:21.862423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:21.883480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94221c3e732d4b6994f330e0320dfb05', 'request_id': 'req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.894438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94221c3e732d4b6994f330e0320dfb05', 'request_id': 'req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.910422 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94221c3e732d4b6994f330e0320dfb05', 'request_id': 'req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.953837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] The token cache is being invalidate because project 94221c3e732d4b6994f330e0320dfb05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:21.955340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.958546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-194bfa8b-3606-4092-a9bc-9b97b45bdb3b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:21.961440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 696/1427] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:21 2022] DELETE /identity/v3/projects/94221c3e732d4b6994f330e0320dfb05 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:21.972533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fd448-9e66-4d92-a99a-a9b80c1920d4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:21.973488 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fd448-9e66-4d92-a99a-a9b80c1920d4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:21.974054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fd448-9e66-4d92-a99a-a9b80c1920d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:21.994295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb295e0-ac36-488a-a665-398e504e7f1b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:22.026012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-8d2fd448-9e66-4d92-a99a-a9b80c1920d4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:22.028761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-8d2fd448-9e66-4d92-a99a-a9b80c1920d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.079675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:22.080745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.081253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.081754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.085083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.145664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 697/1428] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:22.148836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7109508e-a720-4e25-a391-2561ff1a13bf None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.149127 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7109508e-a720-4e25-a391-2561ff1a13bf None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.149410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7109508e-a720-4e25-a391-2561ff1a13bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.182060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:22.183548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:22.192298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7109508e-a720-4e25-a391-2561ff1a13bf None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:22.193442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7109508e-a720-4e25-a391-2561ff1a13bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.204173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 698/1429] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:22.208565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7109508e-a720-4e25-a391-2561ff1a13bf None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.211063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 732/1430] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:22.219416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7109508e-a720-4e25-a391-2561ff1a13bf None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:22.224354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ad4c73-a62a-4dfd-93c3-49d01822c28b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:22.224984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad4c73-a62a-4dfd-93c3-49d01822c28b admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.225262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad4c73-a62a-4dfd-93c3-49d01822c28b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.226192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad4c73-a62a-4dfd-93c3-49d01822c28b admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.226192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ad4c73-a62a-4dfd-93c3-49d01822c28b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:22.226516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ad4c73-a62a-4dfd-93c3-49d01822c28b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:22.228463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ad4c73-a62a-4dfd-93c3-49d01822c28b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:22.244191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 699/1431] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:22.247931 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.259094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b86aa3-62ec-49b2-99b1-72f36c8323e0 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15196e0-2063-4271-b378-c88ca733b88e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.269131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15196e0-2063-4271-b378-c88ca733b88e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.269431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15196e0-2063-4271-b378-c88ca733b88e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.277429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3986c8cb-2ffe-40dc-8d9f-60d8cf7d9d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:22.288288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3986c8cb-2ffe-40dc-8d9f-60d8cf7d9d5c admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.289741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3986c8cb-2ffe-40dc-8d9f-60d8cf7d9d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.289741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3986c8cb-2ffe-40dc-8d9f-60d8cf7d9d5c admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.293377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3986c8cb-2ffe-40dc-8d9f-60d8cf7d9d5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:22.295914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3986c8cb-2ffe-40dc-8d9f-60d8cf7d9d5c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:22.308164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e15196e0-2063-4271-b378-c88ca733b88e None None] MFA Rules not processed for user `ebdef1d69bb14837b9a0c28cbba332fc`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:22.310082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e15196e0-2063-4271-b378-c88ca733b88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.347269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 733/1432] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:22 2022] POST /identity/v3/projects => generated 381 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:22.351607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3986c8cb-2ffe-40dc-8d9f-60d8cf7d9d5c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.432724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 700/1433] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:22 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:22.494412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15196e0-2063-4271-b378-c88ca733b88e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.497007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e15196e0-2063-4271-b378-c88ca733b88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.512788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:22.525109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3986c8cb-2ffe-40dc-8d9f-60d8cf7d9d5c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:22.527615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15196e0-2063-4271-b378-c88ca733b88e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:22.534226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:22.534935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.535290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.535648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] PATH_INFO: `/v3/projects/0216a8024d3146799f1ecf71ee4de0bd` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.537117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0216a8024d3146799f1ecf71ee4de0bd)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:22.539556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:22.541185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5535793f-9ad0-47b4-b8d5-64e87dde2775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:22.541811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5535793f-9ad0-47b4-b8d5-64e87dde2775 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.542091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5535793f-9ad0-47b4-b8d5-64e87dde2775 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.542364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5535793f-9ad0-47b4-b8d5-64e87dde2775 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.542998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5535793f-9ad0-47b4-b8d5-64e87dde2775 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:22.544957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5535793f-9ad0-47b4-b8d5-64e87dde2775 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:22.599298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 734/1434] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:22 2022] POST /identity/v3/users => generated 291 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:22.600643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5535793f-9ad0-47b4-b8d5-64e87dde2775 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.616009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5535793f-9ad0-47b4-b8d5-64e87dde2775 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:22.626961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:22.646411 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0216a8024d3146799f1ecf71ee4de0bd', 'request_id': 'req-17e596a5-5a15-4fd2-a55b-a8d3aa024652'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.659937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0216a8024d3146799f1ecf71ee4de0bd', 'request_id': 'req-17e596a5-5a15-4fd2-a55b-a8d3aa024652'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.679321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0216a8024d3146799f1ecf71ee4de0bd', 'request_id': 'req-17e596a5-5a15-4fd2-a55b-a8d3aa024652'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.705266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] The token cache is being invalidate because project 0216a8024d3146799f1ecf71ee4de0bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:22.705630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.707601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17e596a5-5a15-4fd2-a55b-a8d3aa024652 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.711472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 701/1435] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:22 2022] DELETE /identity/v3/projects/0216a8024d3146799f1ecf71ee4de0bd => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:22.713009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aab799-1e51-435a-8fa4-407c949e7850 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.713449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aab799-1e51-435a-8fa4-407c949e7850 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.713733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aab799-1e51-435a-8fa4-407c949e7850 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.736813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c7aab799-1e51-435a-8fa4-407c949e7850 None None] MFA Rules not processed for user `552c23b8456a40ec8e01297aca7965d3`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:22.738073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c7aab799-1e51-435a-8fa4-407c949e7850 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.756761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:22.757395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.757679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.757958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.758604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:22.760960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:22.792330 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 735/1436] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:22 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:22.792330 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.794283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.811417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f067711e-4175-4613-9176-1440837b4175 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:22.819048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:22.819626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.819848 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.820111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] PATH_INFO: `/v3/users/d489a074003e4f70acba7f849fbd839b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.821442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d489a074003e4f70acba7f849fbd839b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:22.826055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 702/1437] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:22 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:22.826055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aab799-1e51-435a-8fa4-407c949e7850 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.828856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:22.836531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c7aab799-1e51-435a-8fa4-407c949e7850 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.851514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aab799-1e51-435a-8fa4-407c949e7850 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:22.856215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:22.856851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:22.857135 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:22.857412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] PATH_INFO: `/v3/projects/942ec1fa400e4f5fbc8cef75e25775da` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:22.858549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] RBAC: Authorizing `identity:delete_project(project_id=942ec1fa400e4f5fbc8cef75e25775da)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:22.861386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:22.896247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd489a074003e4f70acba7f849fbd839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.904817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd489a074003e4f70acba7f849fbd839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.918678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd489a074003e4f70acba7f849fbd839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.919423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd489a074003e4f70acba7f849fbd839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.920043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd489a074003e4f70acba7f849fbd839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.920739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '942ec1fa400e4f5fbc8cef75e25775da', 'request_id': 'req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.928770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '942ec1fa400e4f5fbc8cef75e25775da', 'request_id': 'req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.937164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '942ec1fa400e4f5fbc8cef75e25775da', 'request_id': 'req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.941184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 736/1438] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:22 2022] DELETE /identity/v3/users/d489a074003e4f70acba7f849fbd839b => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:22.941184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.950442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:22.964631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] The token cache is being invalidate because project 942ec1fa400e4f5fbc8cef75e25775da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:22.966412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.967688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:22.971335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 703/1439] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:22 2022] DELETE /identity/v3/projects/942ec1fa400e4f5fbc8cef75e25775da => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:22.976192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:22.980351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.072964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:23.097276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:23.104047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a106ac1-6a3a-44d3-9d4b-c986f6dead88 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:23.116864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:23.117515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.117796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.117986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f5b935-c2fc-439d-8f8c-a92243d1a513 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:23.119066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] PATH_INFO: `/v3/users/6810668626a44cdaa4f830cb892acadf` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.121819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] RBAC: Authorizing `identity:delete_user(user_id=6810668626a44cdaa4f830cb892acadf)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:23.128999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:23.130985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:23.131789 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.132074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.132400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.140025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:23.144855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:23.181550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 737/1440] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:22 2022] GET /identity/v3/roles => generated 1420 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:23.184615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:23.192192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.220073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32f9fc-a6d4-4dbb-8265-9320efba9fca admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:23.226565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54c5911-d084-450a-a23a-53f93a582c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:23.227635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c5911-d084-450a-a23a-53f93a582c26 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.229482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c5911-d084-450a-a23a-53f93a582c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.229835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c5911-d084-450a-a23a-53f93a582c26 admin admin] PATH_INFO: `/v3/projects/5ef74c507f1746dda37a98f565018c82/users/1cf8c5d7679a4697be2bde56053c8087/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.244071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6810668626a44cdaa4f830cb892acadf'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.255878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6810668626a44cdaa4f830cb892acadf'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.259442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54c5911-d084-450a-a23a-53f93a582c26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ef74c507f1746dda37a98f565018c82, user_id=1cf8c5d7679a4697be2bde56053c8087, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:23.263042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54c5911-d084-450a-a23a-53f93a582c26 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:23.291616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 738/1441] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:23 2022] PUT /identity/v3/projects/5ef74c507f1746dda37a98f565018c82/users/1cf8c5d7679a4697be2bde56053c8087/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:23.294914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9d5052-78bb-4108-a51a-e8b76ffae83f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.295285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9d5052-78bb-4108-a51a-e8b76ffae83f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.295523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9d5052-78bb-4108-a51a-e8b76ffae83f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.308904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6810668626a44cdaa4f830cb892acadf'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.311718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6810668626a44cdaa4f830cb892acadf'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.314644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6810668626a44cdaa4f830cb892acadf'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.330826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 704/1442] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:22 2022] DELETE /identity/v3/users/6810668626a44cdaa4f830cb892acadf => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:23.335750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-df9d5052-78bb-4108-a51a-e8b76ffae83f None None] MFA Rules not processed for user `1cf8c5d7679a4697be2bde56053c8087`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:23.337009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-df9d5052-78bb-4108-a51a-e8b76ffae83f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.340365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:23.342462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.431358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 739/1443] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:23 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:23.438683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9d5052-78bb-4108-a51a-e8b76ffae83f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:23.442911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-df9d5052-78bb-4108-a51a-e8b76ffae83f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.473805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce46a52-a57d-49bb-b76b-44579526b78c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:23.480367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:23.481426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.481614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.481843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] PATH_INFO: `/v3/projects/d3cdd94a225a46a8bddd2fcfe7d42430` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.488073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3cdd94a225a46a8bddd2fcfe7d42430)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:23.488073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:23.569307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3cdd94a225a46a8bddd2fcfe7d42430', 'request_id': 'req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.578232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3cdd94a225a46a8bddd2fcfe7d42430', 'request_id': 'req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.594393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9d5052-78bb-4108-a51a-e8b76ffae83f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:23.605545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3cdd94a225a46a8bddd2fcfe7d42430', 'request_id': 'req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.626476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] The token cache is being invalidate because project d3cdd94a225a46a8bddd2fcfe7d42430 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:23.626895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.628226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f09c64c3-c9a5-4680-a2b5-68f3c3fe5daa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:23.631246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 705/1444] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:23 2022] DELETE /identity/v3/projects/d3cdd94a225a46a8bddd2fcfe7d42430 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:23.637110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a949a-daad-47ee-8e5f-d780fa949a98 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.637412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a949a-daad-47ee-8e5f-d780fa949a98 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.638455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a949a-daad-47ee-8e5f-d780fa949a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.666432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:23.677799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-be6a949a-daad-47ee-8e5f-d780fa949a98 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:23.680975 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-be6a949a-daad-47ee-8e5f-d780fa949a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.706583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:23.707213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.707500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.708429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.709097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:23.713069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:23.723033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:23.763604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 740/1445] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:23 2022] POST /identity/v3/projects => generated 383 bytes in 327 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:23.765547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:23.768831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 706/1446] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:23.770187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.777767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e585e987-7fcf-4b0a-b25f-5638cf8114e7 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.778067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e585e987-7fcf-4b0a-b25f-5638cf8114e7 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.778348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e585e987-7fcf-4b0a-b25f-5638cf8114e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.802496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e585e987-7fcf-4b0a-b25f-5638cf8114e7 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:23.803933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e585e987-7fcf-4b0a-b25f-5638cf8114e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.884476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f52a14-54ae-4ae9-83a7-2ae22070b46e admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:23.889496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 707/1447] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:23.890588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e585e987-7fcf-4b0a-b25f-5638cf8114e7 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:23.892962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e585e987-7fcf-4b0a-b25f-5638cf8114e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:23.905773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c02543-484d-4c90-8ad6-ace8eb4dd073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:23.906870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e585e987-7fcf-4b0a-b25f-5638cf8114e7 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:23.906987 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c02543-484d-4c90-8ad6-ace8eb4dd073 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.908119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c02543-484d-4c90-8ad6-ace8eb4dd073 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.908433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c02543-484d-4c90-8ad6-ace8eb4dd073 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.909085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c02543-484d-4c90-8ad6-ace8eb4dd073 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:23.912122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:23.912795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:23.913081 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:23.913358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] PATH_INFO: `/v3/projects/01578563aeac46149e0b4abe67303101` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:23.914498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01578563aeac46149e0b4abe67303101)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:23.917051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:23.917713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c02543-484d-4c90-8ad6-ace8eb4dd073 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:23.984762 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 741/1448] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:23 2022] POST /identity/v3/users => generated 298 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:23.986074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c02543-484d-4c90-8ad6-ace8eb4dd073 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.011247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c02543-484d-4c90-8ad6-ace8eb4dd073 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.018748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01578563aeac46149e0b4abe67303101', 'request_id': 'req-3e473be8-523b-4e19-8471-20f4b9ce46a3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:24.028741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01578563aeac46149e0b4abe67303101', 'request_id': 'req-3e473be8-523b-4e19-8471-20f4b9ce46a3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:24.043972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01578563aeac46149e0b4abe67303101', 'request_id': 'req-3e473be8-523b-4e19-8471-20f4b9ce46a3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:24.069164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] The token cache is being invalidate because project 01578563aeac46149e0b4abe67303101 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:24.069633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:24.071636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:24.074005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 708/1449] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:23 2022] DELETE /identity/v3/projects/01578563aeac46149e0b4abe67303101 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:24.076844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.079834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:24.161085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:24.173000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece20335-a60e-4529-b418-36ecf5af4325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.174399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece20335-a60e-4529-b418-36ecf5af4325 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.174692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece20335-a60e-4529-b418-36ecf5af4325 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.174974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece20335-a60e-4529-b418-36ecf5af4325 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.176081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece20335-a60e-4529-b418-36ecf5af4325 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:24.176319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece20335-a60e-4529-b418-36ecf5af4325 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.178606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece20335-a60e-4529-b418-36ecf5af4325 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.198691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 742/1450] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:24.200731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e2f9d5-4038-4f35-ac9d-9040f20ade67 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.201104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e2f9d5-4038-4f35-ac9d-9040f20ade67 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.201617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e2f9d5-4038-4f35-ac9d-9040f20ade67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.207185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e473be8-523b-4e19-8471-20f4b9ce46a3 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.219727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.220383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.220665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.220942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.221591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.223386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.231235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-25e2f9d5-4038-4f35-ac9d-9040f20ade67 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:24.232655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-25e2f9d5-4038-4f35-ac9d-9040f20ade67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:24.235103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 709/1451] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:24 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:24.235930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.237681 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:24.246491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 743/1452] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:24.247277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e2f9d5-4038-4f35-ac9d-9040f20ade67 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.258616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ce5b3b-2ae6-45fa-b217-9500082d7d69 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.261674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e2f9d5-4038-4f35-ac9d-9040f20ade67 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.265492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592f5e3c-8c74-4aa0-b245-06d18e6e5647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.266800 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592f5e3c-8c74-4aa0-b245-06d18e6e5647 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.267231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592f5e3c-8c74-4aa0-b245-06d18e6e5647 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.267716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592f5e3c-8c74-4aa0-b245-06d18e6e5647 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.268023 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8afab-1a81-4f2d-a7ed-5f24d8df5025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.268455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592f5e3c-8c74-4aa0-b245-06d18e6e5647 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.268947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8afab-1a81-4f2d-a7ed-5f24d8df5025 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.269299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8afab-1a81-4f2d-a7ed-5f24d8df5025 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.269653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8afab-1a81-4f2d-a7ed-5f24d8df5025 admin admin] PATH_INFO: `/v3/projects/77e592dab1f8417980ae9cd4e6ccb12d/users/595b209ff7a145259b1d4f5cc3d22653/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.270861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592f5e3c-8c74-4aa0-b245-06d18e6e5647 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.290708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc8afab-1a81-4f2d-a7ed-5f24d8df5025 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77e592dab1f8417980ae9cd4e6ccb12d, user_id=595b209ff7a145259b1d4f5cc3d22653, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.296424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc8afab-1a81-4f2d-a7ed-5f24d8df5025 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.310895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 710/1453] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:24 2022] POST /identity/v3/projects => generated 363 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:24.312907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b81f66-dfbd-4f76-a30c-4811cc346324 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.313664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b81f66-dfbd-4f76-a30c-4811cc346324 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.314155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b81f66-dfbd-4f76-a30c-4811cc346324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.315940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 744/1454] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:24 2022] PUT /identity/v3/projects/77e592dab1f8417980ae9cd4e6ccb12d/users/595b209ff7a145259b1d4f5cc3d22653/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:24.317012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8afab-1a81-4f2d-a7ed-5f24d8df5025 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.342099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-27b81f66-dfbd-4f76-a30c-4811cc346324 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:24.344056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-27b81f66-dfbd-4f76-a30c-4811cc346324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:24.381645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:24.410464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8afab-1a81-4f2d-a7ed-5f24d8df5025 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.438787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 711/1455] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:24.440060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b81f66-dfbd-4f76-a30c-4811cc346324 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.455619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b81f66-dfbd-4f76-a30c-4811cc346324 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.463536 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1374b829-9b9e-450b-a290-483661aed05d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.464262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1374b829-9b9e-450b-a290-483661aed05d admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.464542 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1374b829-9b9e-450b-a290-483661aed05d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.464811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1374b829-9b9e-450b-a290-483661aed05d admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.465956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1374b829-9b9e-450b-a290-483661aed05d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.467780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1374b829-9b9e-450b-a290-483661aed05d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.491898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 712/1456] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:24 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:24.493053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1374b829-9b9e-450b-a290-483661aed05d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.509618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68fdcae-ca7e-4287-8356-e8bfd86ac72c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.510633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68fdcae-ca7e-4287-8356-e8bfd86ac72c admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.511154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68fdcae-ca7e-4287-8356-e8bfd86ac72c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.511444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68fdcae-ca7e-4287-8356-e8bfd86ac72c admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.512076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68fdcae-ca7e-4287-8356-e8bfd86ac72c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.514168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68fdcae-ca7e-4287-8356-e8bfd86ac72c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.527738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1374b829-9b9e-450b-a290-483661aed05d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.542427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.543162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.543451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.543923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.544662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:24.545019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.546679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.561313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 745/1457] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:24 2022] POST /identity/v3/users => generated 282 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:24.561313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68fdcae-ca7e-4287-8356-e8bfd86ac72c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.575056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 713/1458] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:24.575056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.575936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68fdcae-ca7e-4287-8356-e8bfd86ac72c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.584198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2672c850-35ea-4c0b-a096-bb0cbddb9eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.586014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2672c850-35ea-4c0b-a096-bb0cbddb9eba admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.586322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2672c850-35ea-4c0b-a096-bb0cbddb9eba admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.586604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2672c850-35ea-4c0b-a096-bb0cbddb9eba admin admin] PATH_INFO: `/v3/projects/77e592dab1f8417980ae9cd4e6ccb12d/users/595b209ff7a145259b1d4f5cc3d22653/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.589724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875092db-3b97-49dd-83d7-a7c503df4d53 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.590129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2672c850-35ea-4c0b-a096-bb0cbddb9eba admin admin] RBAC: Authorizing `identity:create_grant(project_id=77e592dab1f8417980ae9cd4e6ccb12d, user_id=595b209ff7a145259b1d4f5cc3d22653, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.596185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdcc90a-bdf1-45d9-a53b-b341d42bec7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.596605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2672c850-35ea-4c0b-a096-bb0cbddb9eba admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.597042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdcc90a-bdf1-45d9-a53b-b341d42bec7e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.597200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdcc90a-bdf1-45d9-a53b-b341d42bec7e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.597481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdcc90a-bdf1-45d9-a53b-b341d42bec7e admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.598133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdcc90a-bdf1-45d9-a53b-b341d42bec7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.599889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdcc90a-bdf1-45d9-a53b-b341d42bec7e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.621582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 714/1460] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:24 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:24.621582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdcc90a-bdf1-45d9-a53b-b341d42bec7e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.622241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 746/1460] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:24 2022] PUT /identity/v3/projects/77e592dab1f8417980ae9cd4e6ccb12d/users/595b209ff7a145259b1d4f5cc3d22653/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:24.623485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2672c850-35ea-4c0b-a096-bb0cbddb9eba admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.634624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdcc90a-bdf1-45d9-a53b-b341d42bec7e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.744896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c186d-e6de-45c1-9d89-046461a72b0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.745556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c186d-e6de-45c1-9d89-046461a72b0a admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.745843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c186d-e6de-45c1-9d89-046461a72b0a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.746317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c186d-e6de-45c1-9d89-046461a72b0a admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.746964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c186d-e6de-45c1-9d89-046461a72b0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.749886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c186d-e6de-45c1-9d89-046461a72b0a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.751580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2672c850-35ea-4c0b-a096-bb0cbddb9eba admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.757602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4026768-66d6-4067-90ab-3817b8929e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.758291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4026768-66d6-4067-90ab-3817b8929e17 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.758579 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4026768-66d6-4067-90ab-3817b8929e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.758857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4026768-66d6-4067-90ab-3817b8929e17 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.759656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4026768-66d6-4067-90ab-3817b8929e17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.761422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4026768-66d6-4067-90ab-3817b8929e17 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.770248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:24.796674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 747/1461] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:24 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:24.796674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027aec89-df93-433e-84cf-5bf1e45b1e0a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.796674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027aec89-df93-433e-84cf-5bf1e45b1e0a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.796674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027aec89-df93-433e-84cf-5bf1e45b1e0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.809603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 715/1462] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:24 2022] POST /identity/v3/projects => generated 383 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:24.810476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c186d-e6de-45c1-9d89-046461a72b0a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.824246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-027aec89-df93-433e-84cf-5bf1e45b1e0a None None] MFA Rules not processed for user `595b209ff7a145259b1d4f5cc3d22653`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:24.825585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-027aec89-df93-433e-84cf-5bf1e45b1e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:24.880116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:24.882280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c186d-e6de-45c1-9d89-046461a72b0a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.888615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530115d2-2994-476f-a748-226809d273bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.889282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530115d2-2994-476f-a748-226809d273bc admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.889566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530115d2-2994-476f-a748-226809d273bc admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.889840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530115d2-2994-476f-a748-226809d273bc admin admin] PATH_INFO: `/v3/projects/3c744e130db340d489e8d60444f9490e/users/db638e4886ac4982912abb8ab9d317d2/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.904753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 748/1463] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:24 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:24.905343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027aec89-df93-433e-84cf-5bf1e45b1e0a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.909140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530115d2-2994-476f-a748-226809d273bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c744e130db340d489e8d60444f9490e, user_id=db638e4886ac4982912abb8ab9d317d2, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.913564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530115d2-2994-476f-a748-226809d273bc admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.919954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027aec89-df93-433e-84cf-5bf1e45b1e0a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:24.925251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985e8874-a183-4392-aafe-f445b6cda2b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:24.925922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985e8874-a183-4392-aafe-f445b6cda2b5 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.926206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985e8874-a183-4392-aafe-f445b6cda2b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.926531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985e8874-a183-4392-aafe-f445b6cda2b5 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.927150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985e8874-a183-4392-aafe-f445b6cda2b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:24.929306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985e8874-a183-4392-aafe-f445b6cda2b5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:24.931751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 716/1464] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:24 2022] PUT /identity/v3/projects/3c744e130db340d489e8d60444f9490e/users/db638e4886ac4982912abb8ab9d317d2/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:24.941616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530115d2-2994-476f-a748-226809d273bc admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:24.943578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-530115d2-2994-476f-a748-226809d273bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:24.958530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 749/1465] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:24 2022] POST /identity/v3/users => generated 292 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:24.958530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb7504e-1133-49c6-93de-de53ed0db00e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:24.958530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb7504e-1133-49c6-93de-de53ed0db00e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:24.958530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb7504e-1133-49c6-93de-de53ed0db00e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:24.994582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-afb7504e-1133-49c6-93de-de53ed0db00e None None] MFA Rules not processed for user `db638e4886ac4982912abb8ab9d317d2`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:24.997278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-afb7504e-1133-49c6-93de-de53ed0db00e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:25.027367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530115d2-2994-476f-a748-226809d273bc admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.035705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b29d0de-c310-41dc-a4dc-a805a9ee187d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.036818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b29d0de-c310-41dc-a4dc-a805a9ee187d service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.037317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b29d0de-c310-41dc-a4dc-a805a9ee187d service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.037882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b29d0de-c310-41dc-a4dc-a805a9ee187d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.077973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 750/1466] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:24 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:25.078957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb7504e-1133-49c6-93de-de53ed0db00e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.156453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b29d0de-c310-41dc-a4dc-a805a9ee187d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.158157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b29d0de-c310-41dc-a4dc-a805a9ee187d service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.171250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 717/1467] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:24 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:25.176559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b29d0de-c310-41dc-a4dc-a805a9ee187d service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.191016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb7504e-1133-49c6-93de-de53ed0db00e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.198098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b29d0de-c310-41dc-a4dc-a805a9ee187d service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.207039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117ca09-38e3-479e-8103-349e41970abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.208319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117ca09-38e3-479e-8103-349e41970abc admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.209194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117ca09-38e3-479e-8103-349e41970abc admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.209475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117ca09-38e3-479e-8103-349e41970abc admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.210292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117ca09-38e3-479e-8103-349e41970abc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.212683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117ca09-38e3-479e-8103-349e41970abc admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.224592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c301298-2ee2-4671-a171-cf3b4eafd7de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.225285 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c301298-2ee2-4671-a171-cf3b4eafd7de admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.225572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c301298-2ee2-4671-a171-cf3b4eafd7de admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.225855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c301298-2ee2-4671-a171-cf3b4eafd7de admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.227222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c301298-2ee2-4671-a171-cf3b4eafd7de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.229312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 751/1468] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:25 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:25.229817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c301298-2ee2-4671-a171-cf3b4eafd7de admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.233307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117ca09-38e3-479e-8103-349e41970abc admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.264893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117ca09-38e3-479e-8103-349e41970abc admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.275181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e3f4ac-672f-4541-bc43-0d5c4681f6c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.275837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e3f4ac-672f-4541-bc43-0d5c4681f6c8 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.276176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e3f4ac-672f-4541-bc43-0d5c4681f6c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.276512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e3f4ac-672f-4541-bc43-0d5c4681f6c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.277304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e3f4ac-672f-4541-bc43-0d5c4681f6c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.279229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e3f4ac-672f-4541-bc43-0d5c4681f6c8 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.291386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 718/1469] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:25 2022] POST /identity/v3/projects => generated 365 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:25.292817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c301298-2ee2-4671-a171-cf3b4eafd7de admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.309044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 752/1470] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:25 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:25.309044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e3f4ac-672f-4541-bc43-0d5c4681f6c8 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.312014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:25.320325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e3f4ac-672f-4541-bc43-0d5c4681f6c8 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.362185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:25.371918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c301298-2ee2-4671-a171-cf3b4eafd7de admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.378013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb8835-4137-47ad-acff-19fc02675208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.378628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb8835-4137-47ad-acff-19fc02675208 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.378907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb8835-4137-47ad-acff-19fc02675208 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.379174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb8835-4137-47ad-acff-19fc02675208 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.379810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eb8835-4137-47ad-acff-19fc02675208 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.380426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b001c0-a819-479e-844d-4f3b0e8b18b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.381116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b001c0-a819-479e-844d-4f3b0e8b18b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.381399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b001c0-a819-479e-844d-4f3b0e8b18b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.381704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b001c0-a819-479e-844d-4f3b0e8b18b6 admin admin] PATH_INFO: `/v3/projects/bd6764bdd18544a689385cb4a9ce5ed3/users/bfd17a7ae0ae41ed9ca802e18b85e4bc/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.382004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eb8835-4137-47ad-acff-19fc02675208 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.397969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b001c0-a819-479e-844d-4f3b0e8b18b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd6764bdd18544a689385cb4a9ce5ed3, user_id=bfd17a7ae0ae41ed9ca802e18b85e4bc, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.400883 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b001c0-a819-479e-844d-4f3b0e8b18b6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.406556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 719/1471] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:25 2022] POST /identity/v3/users => generated 289 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:25.407606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb8835-4137-47ad-acff-19fc02675208 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.414643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 753/1472] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:25 2022] PUT /identity/v3/projects/bd6764bdd18544a689385cb4a9ce5ed3/users/bfd17a7ae0ae41ed9ca802e18b85e4bc/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:25.415588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b001c0-a819-479e-844d-4f3b0e8b18b6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.467989 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb8835-4137-47ad-acff-19fc02675208 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.478901 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b001c0-a819-479e-844d-4f3b0e8b18b6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.483643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed06812-d73c-46f2-be7d-3d241b46ea16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.484311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed06812-d73c-46f2-be7d-3d241b46ea16 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.484670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed06812-d73c-46f2-be7d-3d241b46ea16 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.484947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed06812-d73c-46f2-be7d-3d241b46ea16 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.485693 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed06812-d73c-46f2-be7d-3d241b46ea16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.487285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed06812-d73c-46f2-be7d-3d241b46ea16 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.501438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 754/1473] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:25 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:25.501438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c470f57-4c46-47e3-8200-0d91a3ea8cf8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.501438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c470f57-4c46-47e3-8200-0d91a3ea8cf8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.501438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c470f57-4c46-47e3-8200-0d91a3ea8cf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.530357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6c470f57-4c46-47e3-8200-0d91a3ea8cf8 None None] MFA Rules not processed for user `bfd17a7ae0ae41ed9ca802e18b85e4bc`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:25.531368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.531555 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.531637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6c470f57-4c46-47e3-8200-0d91a3ea8cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:25.531832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.534179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.536308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:25.623000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.624528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.627075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 755/1474] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:25.627997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c470f57-4c46-47e3-8200-0d91a3ea8cf8 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.639327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 720/1475] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:25 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:25.639959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c470f57-4c46-47e3-8200-0d91a3ea8cf8 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.640090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.646747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606dd921-2755-4fa3-a5ae-3f729c4599b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.647416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606dd921-2755-4fa3-a5ae-3f729c4599b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.648222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606dd921-2755-4fa3-a5ae-3f729c4599b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.648222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606dd921-2755-4fa3-a5ae-3f729c4599b5 admin admin] PATH_INFO: `/v3/projects/37e61ff111a941e78e57281f16baed64/users/69a54f1f4eca49ffaaf29313abdbcfd8/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.653574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362b3047-b8a7-4bdb-aff1-c67903de8b5a service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.659150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d57893-1c23-4d5f-86cc-b740986cd0e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.659783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d57893-1c23-4d5f-86cc-b740986cd0e4 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.660069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d57893-1c23-4d5f-86cc-b740986cd0e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.660657 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d57893-1c23-4d5f-86cc-b740986cd0e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.661307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d57893-1c23-4d5f-86cc-b740986cd0e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.663439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d57893-1c23-4d5f-86cc-b740986cd0e4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.666579 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606dd921-2755-4fa3-a5ae-3f729c4599b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37e61ff111a941e78e57281f16baed64, user_id=69a54f1f4eca49ffaaf29313abdbcfd8, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.668590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:25.672416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606dd921-2755-4fa3-a5ae-3f729c4599b5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.702876 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 756/1476] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:25 2022] PUT /identity/v3/projects/37e61ff111a941e78e57281f16baed64/users/69a54f1f4eca49ffaaf29313abdbcfd8/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:25.709302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606dd921-2755-4fa3-a5ae-3f729c4599b5 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.709947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 721/1477] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:25 2022] POST /identity/v3/projects => generated 383 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:25.714065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d57893-1c23-4d5f-86cc-b740986cd0e4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.743176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:25.779284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606dd921-2755-4fa3-a5ae-3f729c4599b5 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.794992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d57893-1c23-4d5f-86cc-b740986cd0e4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.800824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b0de8-16ef-4b52-8848-a6214d34a2b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.801451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b0de8-16ef-4b52-8848-a6214d34a2b2 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.801767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b0de8-16ef-4b52-8848-a6214d34a2b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.802021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b0de8-16ef-4b52-8848-a6214d34a2b2 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.802679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b0de8-16ef-4b52-8848-a6214d34a2b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.804734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b0de8-16ef-4b52-8848-a6214d34a2b2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.856399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2703f-9b74-4f8a-9beb-c8492d676a91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.857126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2703f-9b74-4f8a-9beb-c8492d676a91 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.857454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2703f-9b74-4f8a-9beb-c8492d676a91 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.857726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2703f-9b74-4f8a-9beb-c8492d676a91 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.858368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b2703f-9b74-4f8a-9beb-c8492d676a91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.860911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b2703f-9b74-4f8a-9beb-c8492d676a91 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.866433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 722/1478] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:25 2022] POST /identity/v3/users => generated 298 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:25.867391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b0de8-16ef-4b52-8848-a6214d34a2b2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.872679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 757/1479] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:25 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:25.877162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2703f-9b74-4f8a-9beb-c8492d676a91 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.886982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b0de8-16ef-4b52-8848-a6214d34a2b2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.888821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2703f-9b74-4f8a-9beb-c8492d676a91 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:25.897133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95c206-a288-4c29-9443-ad82c0cc50c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.898168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95c206-a288-4c29-9443-ad82c0cc50c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.898738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95c206-a288-4c29-9443-ad82c0cc50c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.899952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95c206-a288-4c29-9443-ad82c0cc50c8 admin admin] PATH_INFO: `/v3/projects/37e61ff111a941e78e57281f16baed64/users/69a54f1f4eca49ffaaf29313abdbcfd8/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.902660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282bea97-b5d5-4ce5-b87a-618090041a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:25.903457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a95c206-a288-4c29-9443-ad82c0cc50c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37e61ff111a941e78e57281f16baed64, user_id=69a54f1f4eca49ffaaf29313abdbcfd8, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.903585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bea97-b5d5-4ce5-b87a-618090041a04 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.903585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bea97-b5d5-4ce5-b87a-618090041a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.903797 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bea97-b5d5-4ce5-b87a-618090041a04 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:25.904579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282bea97-b5d5-4ce5-b87a-618090041a04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:25.907311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282bea97-b5d5-4ce5-b87a-618090041a04 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.908750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a95c206-a288-4c29-9443-ad82c0cc50c8 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:25.922614 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 758/1480] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:25 2022] PUT /identity/v3/projects/37e61ff111a941e78e57281f16baed64/users/69a54f1f4eca49ffaaf29313abdbcfd8/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:25.924718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 723/1481] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:25 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:25.928840 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95c206-a288-4c29-9443-ad82c0cc50c8 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:25.936738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c62b090-e695-46e4-8d54-fd899b5d2a28 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:25.936983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c62b090-e695-46e4-8d54-fd899b5d2a28 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:25.937255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c62b090-e695-46e4-8d54-fd899b5d2a28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.018176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-6c62b090-e695-46e4-8d54-fd899b5d2a28 None None] MFA Rules not processed for user `69a54f1f4eca49ffaaf29313abdbcfd8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:26.019389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6c62b090-e695-46e4-8d54-fd899b5d2a28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:26.068148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95c206-a288-4c29-9443-ad82c0cc50c8 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:26.090083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97915d5c-ae42-4685-8a6d-07a803106a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:26.090771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97915d5c-ae42-4685-8a6d-07a803106a73 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:26.091051 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97915d5c-ae42-4685-8a6d-07a803106a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:26.091328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97915d5c-ae42-4685-8a6d-07a803106a73 admin admin] PATH_INFO: `/v3/projects/b2c790463a4340cebb5f406aa048e55e/users/5af8a51fce514f8995a8fe3546e43139/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.139529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97915d5c-ae42-4685-8a6d-07a803106a73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2c790463a4340cebb5f406aa048e55e, user_id=5af8a51fce514f8995a8fe3546e43139, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:26.143653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97915d5c-ae42-4685-8a6d-07a803106a73 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:26.210874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 759/1482] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:25 2022] PUT /identity/v3/projects/b2c790463a4340cebb5f406aa048e55e/users/5af8a51fce514f8995a8fe3546e43139/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:26.213105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97915d5c-ae42-4685-8a6d-07a803106a73 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:26.216177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 724/1483] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:25 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:26.230240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c62b090-e695-46e4-8d54-fd899b5d2a28 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:26.266006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:26.282713 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:26.287475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c62b090-e695-46e4-8d54-fd899b5d2a28 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:26.290489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97915d5c-ae42-4685-8a6d-07a803106a73 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:26.295279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805e4c7c-a55a-4f8f-b3f5-2e1bb4405c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:26.295888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e4c7c-a55a-4f8f-b3f5-2e1bb4405c19 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:26.296197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e4c7c-a55a-4f8f-b3f5-2e1bb4405c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:26.296477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e4c7c-a55a-4f8f-b3f5-2e1bb4405c19 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.297685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805e4c7c-a55a-4f8f-b3f5-2e1bb4405c19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:26.299547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805e4c7c-a55a-4f8f-b3f5-2e1bb4405c19 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:26.311040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 760/1484] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:26 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:26.315661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805e4c7c-a55a-4f8f-b3f5-2e1bb4405c19 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:26.325706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805e4c7c-a55a-4f8f-b3f5-2e1bb4405c19 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:26.330712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88043ae-493c-4ff1-84e3-d60d06690486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:26.331372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88043ae-493c-4ff1-84e3-d60d06690486 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:26.331653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88043ae-493c-4ff1-84e3-d60d06690486 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:26.331929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88043ae-493c-4ff1-84e3-d60d06690486 admin admin] PATH_INFO: `/v3/projects/b2c790463a4340cebb5f406aa048e55e/users/5af8a51fce514f8995a8fe3546e43139/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.333874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88043ae-493c-4ff1-84e3-d60d06690486 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2c790463a4340cebb5f406aa048e55e, user_id=5af8a51fce514f8995a8fe3546e43139, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:26.336539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88043ae-493c-4ff1-84e3-d60d06690486 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:26.345015 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57bdb9-40fc-4059-a3b3-8de86059117a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:26.345626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57bdb9-40fc-4059-a3b3-8de86059117a service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:26.346032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57bdb9-40fc-4059-a3b3-8de86059117a service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:26.346312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57bdb9-40fc-4059-a3b3-8de86059117a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.348254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 761/1485] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:26 2022] PUT /identity/v3/projects/b2c790463a4340cebb5f406aa048e55e/users/5af8a51fce514f8995a8fe3546e43139/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:26.362376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f2b52c-0f23-408e-885b-769dbbe9d572 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:26.362818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f2b52c-0f23-408e-885b-769dbbe9d572 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:26.363223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f2b52c-0f23-408e-885b-769dbbe9d572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.409492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-67f2b52c-0f23-408e-885b-769dbbe9d572 None None] MFA Rules not processed for user `5af8a51fce514f8995a8fe3546e43139`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:26.410889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-67f2b52c-0f23-408e-885b-769dbbe9d572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:26.419270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b57bdb9-40fc-4059-a3b3-8de86059117a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:26.420829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b57bdb9-40fc-4059-a3b3-8de86059117a service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:26.431120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 725/1486] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:26 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:26.482595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 762/1487] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:26 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:26.495535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57bdb9-40fc-4059-a3b3-8de86059117a service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:26.504753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57bdb9-40fc-4059-a3b3-8de86059117a service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:26.509348 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f2b52c-0f23-408e-885b-769dbbe9d572 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:26.510887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-67f2b52c-0f23-408e-885b-769dbbe9d572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:26.556634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa8ad01-0752-4e89-93ef-fa2c99862d82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:26.557116 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8ad01-0752-4e89-93ef-fa2c99862d82 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:26.557403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8ad01-0752-4e89-93ef-fa2c99862d82 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:26.557681 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8ad01-0752-4e89-93ef-fa2c99862d82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.567805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa8ad01-0752-4e89-93ef-fa2c99862d82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:26.569309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa8ad01-0752-4e89-93ef-fa2c99862d82 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:26.572819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f2b52c-0f23-408e-885b-769dbbe9d572 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:26.576439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83809139-fa6e-4ce8-b87b-412eda6d6618 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:26.577042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83809139-fa6e-4ce8-b87b-412eda6d6618 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:26.577321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83809139-fa6e-4ce8-b87b-412eda6d6618 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:26.577592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83809139-fa6e-4ce8-b87b-412eda6d6618 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.580601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 726/1488] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:26 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:26.644029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83809139-fa6e-4ce8-b87b-412eda6d6618 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:26.645498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83809139-fa6e-4ce8-b87b-412eda6d6618 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:26.655458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 763/1489] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:26 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:26.700381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa8ad01-0752-4e89-93ef-fa2c99862d82 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:26.709361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa8ad01-0752-4e89-93ef-fa2c99862d82 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:26.714026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ec4046-78d2-44dd-a1ef-19fe2c3f522f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:26.715079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec4046-78d2-44dd-a1ef-19fe2c3f522f service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:26.715546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec4046-78d2-44dd-a1ef-19fe2c3f522f service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:26.715984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec4046-78d2-44dd-a1ef-19fe2c3f522f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:26.800126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:26.803592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ec4046-78d2-44dd-a1ef-19fe2c3f522f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:26.805385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ec4046-78d2-44dd-a1ef-19fe2c3f522f service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:26.817473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 727/1490] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:27.101336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83809139-fa6e-4ce8-b87b-412eda6d6618 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:27.105466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:27.169173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83809139-fa6e-4ce8-b87b-412eda6d6618 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:27.174524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:27.175147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:27.175427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:27.175696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] PATH_INFO: `/v3/users/bfd17a7ae0ae41ed9ca802e18b85e4bc` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:27.176834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bfd17a7ae0ae41ed9ca802e18b85e4bc)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:27.178794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:27.226759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bfd17a7ae0ae41ed9ca802e18b85e4bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.233641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bfd17a7ae0ae41ed9ca802e18b85e4bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.249327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bfd17a7ae0ae41ed9ca802e18b85e4bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.250090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bfd17a7ae0ae41ed9ca802e18b85e4bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.250704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bfd17a7ae0ae41ed9ca802e18b85e4bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.267438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 764/1491] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:27 2022] DELETE /identity/v3/users/bfd17a7ae0ae41ed9ca802e18b85e4bc => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:27.269405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3c28a-491d-444d-87f8-a9cc7cda19c4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:27.269900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3c28a-491d-444d-87f8-a9cc7cda19c4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:27.270048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3c28a-491d-444d-87f8-a9cc7cda19c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:27.290726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-70a3c28a-491d-444d-87f8-a9cc7cda19c4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:27.291933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-70a3c28a-491d-444d-87f8-a9cc7cda19c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:27.355132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 728/1492] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:27.365338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:27.367007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:27.424366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3bd5a2-a440-46b3-8a35-33f2be8557a0 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:27.428000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b23989b-044c-4067-ae3d-24c25ad19be9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:27.428632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b23989b-044c-4067-ae3d-24c25ad19be9 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:27.428909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b23989b-044c-4067-ae3d-24c25ad19be9 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:27.429181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b23989b-044c-4067-ae3d-24c25ad19be9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:27.439694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b23989b-044c-4067-ae3d-24c25ad19be9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:27.441162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b23989b-044c-4067-ae3d-24c25ad19be9 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:27.451708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 765/1493] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:27.733488 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a3c28a-491d-444d-87f8-a9cc7cda19c4 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:27.734995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-70a3c28a-491d-444d-87f8-a9cc7cda19c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:27.747366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a3c28a-491d-444d-87f8-a9cc7cda19c4 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:27.752690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:27.753324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:27.753634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:27.753907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] PATH_INFO: `/v3/projects/bd6764bdd18544a689385cb4a9ce5ed3` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:27.755034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd6764bdd18544a689385cb4a9ce5ed3)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:27.756980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:27.805846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd6764bdd18544a689385cb4a9ce5ed3', 'request_id': 'req-11058ef2-972b-4dc1-ac9d-138cb3d467cf'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.811280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd6764bdd18544a689385cb4a9ce5ed3', 'request_id': 'req-11058ef2-972b-4dc1-ac9d-138cb3d467cf'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.819133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd6764bdd18544a689385cb4a9ce5ed3', 'request_id': 'req-11058ef2-972b-4dc1-ac9d-138cb3d467cf'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.833582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] The token cache is being invalidate because project bd6764bdd18544a689385cb4a9ce5ed3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:27.833862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.834510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.836051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 729/1494] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:27 2022] DELETE /identity/v3/projects/bd6764bdd18544a689385cb4a9ce5ed3 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:27.840041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b23989b-044c-4067-ae3d-24c25ad19be9 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:27.849836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b23989b-044c-4067-ae3d-24c25ad19be9 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:27.869163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:27.907538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:27.908187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:27.908482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:27.908760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] PATH_INFO: `/v3/users/5af8a51fce514f8995a8fe3546e43139` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:27.909875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5af8a51fce514f8995a8fe3546e43139)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:27.911776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:27.960755 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5af8a51fce514f8995a8fe3546e43139'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.968470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5af8a51fce514f8995a8fe3546e43139'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.984174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5af8a51fce514f8995a8fe3546e43139'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.985424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5af8a51fce514f8995a8fe3546e43139'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:27.986561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d6fc2dea-8edc-4698-9840-e43225d2dfa9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5af8a51fce514f8995a8fe3546e43139'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:28.011007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 766/1495] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:27 2022] DELETE /identity/v3/users/5af8a51fce514f8995a8fe3546e43139 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:28.367571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:28.369089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:28.405999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11058ef2-972b-4dc1-ac9d-138cb3d467cf admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:28.467276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:28.467916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.468268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.470529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] PATH_INFO: `/v3/projects/b2c790463a4340cebb5f406aa048e55e` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:28.472112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b2c790463a4340cebb5f406aa048e55e)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:28.474492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:28.500510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:28.533151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b2c790463a4340cebb5f406aa048e55e', 'request_id': 'req-86996042-7ecd-46ad-999a-e20d988be028'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:28.540080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b2c790463a4340cebb5f406aa048e55e', 'request_id': 'req-86996042-7ecd-46ad-999a-e20d988be028'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:28.547909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b2c790463a4340cebb5f406aa048e55e', 'request_id': 'req-86996042-7ecd-46ad-999a-e20d988be028'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:28.562848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] The token cache is being invalidate because project b2c790463a4340cebb5f406aa048e55e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:28.563156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:28.563814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-86996042-7ecd-46ad-999a-e20d988be028 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:28.565419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 730/1496] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:28 2022] DELETE /identity/v3/projects/b2c790463a4340cebb5f406aa048e55e => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:28.570413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9264f63-2d09-4049-881d-67d17d51df90 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.570762 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9264f63-2d09-4049-881d-67d17d51df90 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.571135 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9264f63-2d09-4049-881d-67d17d51df90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:28.592625 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e9264f63-2d09-4049-881d-67d17d51df90 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:28.594362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e9264f63-2d09-4049-881d-67d17d51df90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:28.660644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 767/1497] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:28.668685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3cae62-51a4-4363-9783-7d4b402ad458 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.668977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3cae62-51a4-4363-9783-7d4b402ad458 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.669255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3cae62-51a4-4363-9783-7d4b402ad458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:28.689910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-1a3cae62-51a4-4363-9783-7d4b402ad458 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:28.691119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1a3cae62-51a4-4363-9783-7d4b402ad458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:28.700069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 731/1498] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:28.704476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9264f63-2d09-4049-881d-67d17d51df90 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:28.714298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9264f63-2d09-4049-881d-67d17d51df90 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:28.719374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:28.720048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.720366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.720647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:28.721288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:28.721593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:28.723187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:28.735512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 768/1499] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:28.739811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3cae62-51a4-4363-9783-7d4b402ad458 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:28.749539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3cae62-51a4-4363-9783-7d4b402ad458 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:28.754423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e6fe3a-e4c7-4e17-b67b-2fa20e77539b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:28.755010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e6fe3a-e4c7-4e17-b67b-2fa20e77539b admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.755293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e6fe3a-e4c7-4e17-b67b-2fa20e77539b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.755599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e6fe3a-e4c7-4e17-b67b-2fa20e77539b admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:28.756343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e6fe3a-e4c7-4e17-b67b-2fa20e77539b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:28.759385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e6fe3a-e4c7-4e17-b67b-2fa20e77539b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:28.789674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 732/1500] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:28 2022] POST /identity/v3/projects => generated 375 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:28.794019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:28.804258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a854da-5862-4f9e-a821-710ddb289dd7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:28.834242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:28.869661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6fb51-fe9d-4574-b050-fbe612ab2635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:28.870397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6fb51-fe9d-4574-b050-fbe612ab2635 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.870541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6fb51-fe9d-4574-b050-fbe612ab2635 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.870811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6fb51-fe9d-4574-b050-fbe612ab2635 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:28.871463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6fb51-fe9d-4574-b050-fbe612ab2635 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:28.873439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6fb51-fe9d-4574-b050-fbe612ab2635 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:28.897528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 769/1501] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:28 2022] POST /identity/v3/users => generated 288 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:28.903780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e6fe3a-e4c7-4e17-b67b-2fa20e77539b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:28.916428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e6fe3a-e4c7-4e17-b67b-2fa20e77539b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:28.920956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c2881-39f7-4650-a333-944d8c9f1f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:28.921556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c2881-39f7-4650-a333-944d8c9f1f90 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.921868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c2881-39f7-4650-a333-944d8c9f1f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.922617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c2881-39f7-4650-a333-944d8c9f1f90 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:28.923252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5c2881-39f7-4650-a333-944d8c9f1f90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:28.924909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5c2881-39f7-4650-a333-944d8c9f1f90 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:28.934740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 733/1502] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:28.939750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6fb51-fe9d-4574-b050-fbe612ab2635 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:28.950491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6fb51-fe9d-4574-b050-fbe612ab2635 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:28.956211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a558cb-12a3-4436-9c29-2629125f8b44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:28.956813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a558cb-12a3-4436-9c29-2629125f8b44 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.957093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a558cb-12a3-4436-9c29-2629125f8b44 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.957365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a558cb-12a3-4436-9c29-2629125f8b44 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:28.958006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a558cb-12a3-4436-9c29-2629125f8b44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:28.959701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a558cb-12a3-4436-9c29-2629125f8b44 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:28.975054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 770/1503] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:28 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:28.979839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c2881-39f7-4650-a333-944d8c9f1f90 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:28.991158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c2881-39f7-4650-a333-944d8c9f1f90 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:28.996071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af77ff3-49be-4bf8-a9e1-bd9e12441045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:28.996773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af77ff3-49be-4bf8-a9e1-bd9e12441045 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:28.997095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af77ff3-49be-4bf8-a9e1-bd9e12441045 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:28.997406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af77ff3-49be-4bf8-a9e1-bd9e12441045 admin admin] PATH_INFO: `/v3/projects/4f147e9a2fab4213a3be34c806c0d131/users/5ad39c30cfb04e5fa36f6038a6011afd/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.013385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af77ff3-49be-4bf8-a9e1-bd9e12441045 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f147e9a2fab4213a3be34c806c0d131, user_id=5ad39c30cfb04e5fa36f6038a6011afd, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:29.016667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af77ff3-49be-4bf8-a9e1-bd9e12441045 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:29.030314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 734/1504] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:28 2022] PUT /identity/v3/projects/4f147e9a2fab4213a3be34c806c0d131/users/5ad39c30cfb04e5fa36f6038a6011afd/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:29.044065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8834aed9-4ea3-46f4-ad53-4a849bd05ea9 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:29.044405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8834aed9-4ea3-46f4-ad53-4a849bd05ea9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:29.044682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8834aed9-4ea3-46f4-ad53-4a849bd05ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.200015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-8834aed9-4ea3-46f4-ad53-4a849bd05ea9 None None] MFA Rules not processed for user `5ad39c30cfb04e5fa36f6038a6011afd`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:29.201292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8834aed9-4ea3-46f4-ad53-4a849bd05ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:29.287262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 771/1505] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:29.291534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af77ff3-49be-4bf8-a9e1-bd9e12441045 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:29.365620 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af77ff3-49be-4bf8-a9e1-bd9e12441045 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:29.370709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61821485-43c2-482a-a43a-a6eb76093ef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:29.371317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61821485-43c2-482a-a43a-a6eb76093ef5 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:29.371595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61821485-43c2-482a-a43a-a6eb76093ef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:29.371865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61821485-43c2-482a-a43a-a6eb76093ef5 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.372531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61821485-43c2-482a-a43a-a6eb76093ef5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:29.374702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61821485-43c2-482a-a43a-a6eb76093ef5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:29.401760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 735/1506] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:29 2022] POST /identity/v3/projects => generated 375 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:29.406013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8834aed9-4ea3-46f4-ad53-4a849bd05ea9 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:29.417139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8834aed9-4ea3-46f4-ad53-4a849bd05ea9 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:29.474607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1815e358-0c78-416d-a5a7-341bdd568b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:29.475203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1815e358-0c78-416d-a5a7-341bdd568b6b admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:29.475477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1815e358-0c78-416d-a5a7-341bdd568b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:29.475740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1815e358-0c78-416d-a5a7-341bdd568b6b admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.476395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1815e358-0c78-416d-a5a7-341bdd568b6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:29.478267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1815e358-0c78-416d-a5a7-341bdd568b6b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:29.499613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 772/1507] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:29 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:29.503730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61821485-43c2-482a-a43a-a6eb76093ef5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:29.514507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61821485-43c2-482a-a43a-a6eb76093ef5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:29.520220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad81ba99-1519-4b44-bde7-c4854832272c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:29.520837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad81ba99-1519-4b44-bde7-c4854832272c admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:29.521138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad81ba99-1519-4b44-bde7-c4854832272c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:29.521449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad81ba99-1519-4b44-bde7-c4854832272c admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.522090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad81ba99-1519-4b44-bde7-c4854832272c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:29.523704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad81ba99-1519-4b44-bde7-c4854832272c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:29.535094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 736/1508] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:29.541214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1815e358-0c78-416d-a5a7-341bdd568b6b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:29.560553 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1815e358-0c78-416d-a5a7-341bdd568b6b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:29.573734 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4759e983-41a6-4d18-9886-cb4a1f2b5338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:29.574409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4759e983-41a6-4d18-9886-cb4a1f2b5338 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:29.574694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4759e983-41a6-4d18-9886-cb4a1f2b5338 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:29.574965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4759e983-41a6-4d18-9886-cb4a1f2b5338 admin admin] PATH_INFO: `/v3/projects/8fcd433c3694406cbb06fa2d7d301e87/users/0511eea01d36405c89809e5465021aa9/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.600920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4759e983-41a6-4d18-9886-cb4a1f2b5338 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fcd433c3694406cbb06fa2d7d301e87, user_id=0511eea01d36405c89809e5465021aa9, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:29.604583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4759e983-41a6-4d18-9886-cb4a1f2b5338 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:29.621647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 773/1509] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:29 2022] PUT /identity/v3/projects/8fcd433c3694406cbb06fa2d7d301e87/users/0511eea01d36405c89809e5465021aa9/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:29.625628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad81ba99-1519-4b44-bde7-c4854832272c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:29.639403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad81ba99-1519-4b44-bde7-c4854832272c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:29.661918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:29.714646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f35a3ee-8b62-4751-8513-ea9a98dbeb07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:29.715296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35a3ee-8b62-4751-8513-ea9a98dbeb07 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:29.715582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35a3ee-8b62-4751-8513-ea9a98dbeb07 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:29.715848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35a3ee-8b62-4751-8513-ea9a98dbeb07 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.716889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f35a3ee-8b62-4751-8513-ea9a98dbeb07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:29.719194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f35a3ee-8b62-4751-8513-ea9a98dbeb07 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:29.734063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 737/1510] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:29 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:29.736070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4759e983-41a6-4d18-9886-cb4a1f2b5338 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:29.746192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:29.751997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4759e983-41a6-4d18-9886-cb4a1f2b5338 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:29.759657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:29.760659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:29.761227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:29.761514 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] PATH_INFO: `/v3/projects/8fcd433c3694406cbb06fa2d7d301e87/users/0511eea01d36405c89809e5465021aa9/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.763610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fcd433c3694406cbb06fa2d7d301e87, user_id=0511eea01d36405c89809e5465021aa9, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:29.767432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:29.779983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 774/1511] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:29 2022] PUT /identity/v3/projects/8fcd433c3694406cbb06fa2d7d301e87/users/0511eea01d36405c89809e5465021aa9/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:29.790823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4c359a-c04a-4917-914c-9da197897814 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:29.791124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4c359a-c04a-4917-914c-9da197897814 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:29.791373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4c359a-c04a-4917-914c-9da197897814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:29.828911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2b4c359a-c04a-4917-914c-9da197897814 None None] MFA Rules not processed for user `0511eea01d36405c89809e5465021aa9`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:29.830828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2b4c359a-c04a-4917-914c-9da197897814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:29.925092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:29.926197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 738/1512] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:29 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:29.927255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:30.017236 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8db717-392a-4d7d-ad8e-53420430365f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:30.025904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f755619-ac06-4846-b90f-4f97e0962214 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:30.026991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f755619-ac06-4846-b90f-4f97e0962214 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:30.027415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f755619-ac06-4846-b90f-4f97e0962214 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:30.027882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f755619-ac06-4846-b90f-4f97e0962214 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:30.101289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f755619-ac06-4846-b90f-4f97e0962214 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:30.102718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f755619-ac06-4846-b90f-4f97e0962214 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:30.119096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 775/1513] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:33.837886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4c359a-c04a-4917-914c-9da197897814 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:33.839433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2b4c359a-c04a-4917-914c-9da197897814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:33.918392 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4c359a-c04a-4917-914c-9da197897814 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:33.923387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:33.924018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:33.924327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:33.924605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] PATH_INFO: `/v3/users/1cf8c5d7679a4697be2bde56053c8087` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:33.925732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1cf8c5d7679a4697be2bde56053c8087)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:33.927632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:33.982406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:33.986872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1cf8c5d7679a4697be2bde56053c8087'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:33.995633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1cf8c5d7679a4697be2bde56053c8087'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.008838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1cf8c5d7679a4697be2bde56053c8087'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.009502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1cf8c5d7679a4697be2bde56053c8087'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.010123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1cf8c5d7679a4697be2bde56053c8087'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.024162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 739/1514] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:33 2022] DELETE /identity/v3/users/1cf8c5d7679a4697be2bde56053c8087 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:34.029346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080e4de-ae9f-43b6-b0d2-facb50a86c04 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:34.029813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080e4de-ae9f-43b6-b0d2-facb50a86c04 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:34.029917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080e4de-ae9f-43b6-b0d2-facb50a86c04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:34.052258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5080e4de-ae9f-43b6-b0d2-facb50a86c04 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:34.053460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5080e4de-ae9f-43b6-b0d2-facb50a86c04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:34.119118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 776/1515] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:34.131070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:34.131199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5080e4de-ae9f-43b6-b0d2-facb50a86c04 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:34.132524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5080e4de-ae9f-43b6-b0d2-facb50a86c04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:34.132630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:34.149050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afcb79e-48ae-486d-875b-6b6352831aa1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:34.155151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:34.157403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:34.157611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:34.158273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] PATH_INFO: `/v3/users/5ad39c30cfb04e5fa36f6038a6011afd` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:34.159861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ad39c30cfb04e5fa36f6038a6011afd)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:34.162020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:34.204455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:34.212874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5080e4de-ae9f-43b6-b0d2-facb50a86c04 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:34.220167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4a4c94-db02-45c4-8ea6-61062e2c02ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:34.222010 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4a4c94-db02-45c4-8ea6-61062e2c02ab service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:34.222412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4a4c94-db02-45c4-8ea6-61062e2c02ab service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:34.222812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4a4c94-db02-45c4-8ea6-61062e2c02ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:34.228114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ad39c30cfb04e5fa36f6038a6011afd'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.233639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4a4c94-db02-45c4-8ea6-61062e2c02ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:34.235257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4a4c94-db02-45c4-8ea6-61062e2c02ab service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:34.239734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ad39c30cfb04e5fa36f6038a6011afd'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.245881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 777/1516] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:34.258264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ad39c30cfb04e5fa36f6038a6011afd'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.259052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ad39c30cfb04e5fa36f6038a6011afd'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.259738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ad39c30cfb04e5fa36f6038a6011afd'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.276356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 740/1517] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:34 2022] DELETE /identity/v3/users/5ad39c30cfb04e5fa36f6038a6011afd => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:34.278533 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4f22c-fa13-4a20-8b44-436b3c25adf2 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:34.278820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4f22c-fa13-4a20-8b44-436b3c25adf2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:34.279150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4f22c-fa13-4a20-8b44-436b3c25adf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:34.301171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-bfc4f22c-fa13-4a20-8b44-436b3c25adf2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:34.302895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bfc4f22c-fa13-4a20-8b44-436b3c25adf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:34.382538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 778/1518] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:34.393189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:34.394748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:34.466440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc7fe5c-579f-4414-b236-5a9c5cb4052d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:34.470585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d13e1a-447c-4f35-9bc7-dbdb4a0fb62b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:34.471216 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d13e1a-447c-4f35-9bc7-dbdb4a0fb62b service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:34.471496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d13e1a-447c-4f35-9bc7-dbdb4a0fb62b service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:34.471766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d13e1a-447c-4f35-9bc7-dbdb4a0fb62b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:34.481355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d13e1a-447c-4f35-9bc7-dbdb4a0fb62b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:34.482780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d13e1a-447c-4f35-9bc7-dbdb4a0fb62b service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:34.494246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 741/1519] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:34.642245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4f22c-fa13-4a20-8b44-436b3c25adf2 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:34.643721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bfc4f22c-fa13-4a20-8b44-436b3c25adf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:34.659559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4f22c-fa13-4a20-8b44-436b3c25adf2 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:34.684199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:34.686681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:34.687252 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:34.687919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] PATH_INFO: `/v3/projects/5ef74c507f1746dda37a98f565018c82` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:34.690668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ef74c507f1746dda37a98f565018c82)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:34.696286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:34.772165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ef74c507f1746dda37a98f565018c82', 'request_id': 'req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.779458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ef74c507f1746dda37a98f565018c82', 'request_id': 'req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.790597 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ef74c507f1746dda37a98f565018c82', 'request_id': 'req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.803790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] The token cache is being invalidate because project 5ef74c507f1746dda37a98f565018c82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:34.804080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.804764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.806340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 779/1520] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:34 2022] DELETE /identity/v3/projects/5ef74c507f1746dda37a98f565018c82 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:34.810584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d13e1a-447c-4f35-9bc7-dbdb4a0fb62b service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:34.822301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d13e1a-447c-4f35-9bc7-dbdb4a0fb62b service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:34.862982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:34.892340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:34.893752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:34.897696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:34.898333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:34.898656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:34.898938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] PATH_INFO: `/v3/users/595b209ff7a145259b1d4f5cc3d22653` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:34.900114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] RBAC: Authorizing `identity:delete_user(user_id=595b209ff7a145259b1d4f5cc3d22653)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:34.902123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:34.909049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b0958-6bfd-42c0-a81c-1afe1a5692ed admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:34.913858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:34.914574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:34.914918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:34.915341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] PATH_INFO: `/v3/projects/4f147e9a2fab4213a3be34c806c0d131` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:34.916510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f147e9a2fab4213a3be34c806c0d131)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:34.918437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:34.961795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '595b209ff7a145259b1d4f5cc3d22653'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.974061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '595b209ff7a145259b1d4f5cc3d22653'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.977165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f147e9a2fab4213a3be34c806c0d131', 'request_id': 'req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.987888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f147e9a2fab4213a3be34c806c0d131', 'request_id': 'req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.998025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '595b209ff7a145259b1d4f5cc3d22653'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.998213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f147e9a2fab4213a3be34c806c0d131', 'request_id': 'req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.998933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '595b209ff7a145259b1d4f5cc3d22653'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:34.999524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '595b209ff7a145259b1d4f5cc3d22653'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.005965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:35.016942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 742/1521] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:34 2022] DELETE /identity/v3/users/595b209ff7a145259b1d4f5cc3d22653 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:35.017281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] The token cache is being invalidate because project 4f147e9a2fab4213a3be34c806c0d131 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:35.017576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.018506 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.020188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 780/1522] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:34 2022] DELETE /identity/v3/projects/4f147e9a2fab4213a3be34c806c0d131 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:35.023987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:35.025872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:35.107540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0d87c-77fc-45c8-b514-c6f2d096828d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:35.113144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:35.113968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:35.114201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:35.114511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] PATH_INFO: `/v3/users/0511eea01d36405c89809e5465021aa9` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:35.116141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0511eea01d36405c89809e5465021aa9)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:35.118124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:35.166017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0511eea01d36405c89809e5465021aa9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.172858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0511eea01d36405c89809e5465021aa9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.173973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:35.192143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0511eea01d36405c89809e5465021aa9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.192910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0511eea01d36405c89809e5465021aa9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.193689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0511eea01d36405c89809e5465021aa9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.209207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 743/1523] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:35 2022] DELETE /identity/v3/users/0511eea01d36405c89809e5465021aa9 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:35.474305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:35.476296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:35.564766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce21bfb-e0a0-4cc2-bea4-60af8a596f9d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:35.635544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:35.636272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:35.636554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:35.636826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] PATH_INFO: `/v3/projects/77e592dab1f8417980ae9cd4e6ccb12d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:35.637944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77e592dab1f8417980ae9cd4e6ccb12d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:35.641106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:35.704829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:35.706394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:35.722766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b565d6-e8b3-41c7-b2b6-39d9b1332724 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:35.725133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77e592dab1f8417980ae9cd4e6ccb12d', 'request_id': 'req-6ecdb5c4-2c2b-4943-9440-901cb8491da5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.732653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77e592dab1f8417980ae9cd4e6ccb12d', 'request_id': 'req-6ecdb5c4-2c2b-4943-9440-901cb8491da5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.742393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77e592dab1f8417980ae9cd4e6ccb12d', 'request_id': 'req-6ecdb5c4-2c2b-4943-9440-901cb8491da5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.753797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] The token cache is being invalidate because project 77e592dab1f8417980ae9cd4e6ccb12d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:35.754093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.754772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6ecdb5c4-2c2b-4943-9440-901cb8491da5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.756642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 781/1524] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:35 2022] DELETE /identity/v3/projects/77e592dab1f8417980ae9cd4e6ccb12d => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:35.761189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dedc6b6-5863-4015-8817-6ef53d3700b7 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:35.761556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dedc6b6-5863-4015-8817-6ef53d3700b7 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:35.761835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dedc6b6-5863-4015-8817-6ef53d3700b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:35.783480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-9dedc6b6-5863-4015-8817-6ef53d3700b7 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:35.784721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9dedc6b6-5863-4015-8817-6ef53d3700b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:35.790787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:35.791510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:35.791864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:35.792282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] PATH_INFO: `/v3/projects/8fcd433c3694406cbb06fa2d7d301e87` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:35.793492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fcd433c3694406cbb06fa2d7d301e87)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:35.796161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:35.824402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:35.848388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fcd433c3694406cbb06fa2d7d301e87', 'request_id': 'req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.853970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fcd433c3694406cbb06fa2d7d301e87', 'request_id': 'req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.860708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fcd433c3694406cbb06fa2d7d301e87', 'request_id': 'req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.873121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 782/1525] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:35.874130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] The token cache is being invalidate because project 8fcd433c3694406cbb06fa2d7d301e87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:35.874559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.875697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a1e8de77-f200-4b36-9eed-bda6e4fd0f8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:35.877516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 744/1526] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:35 2022] DELETE /identity/v3/projects/8fcd433c3694406cbb06fa2d7d301e87 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:35.881552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c521b9b1-c850-49bf-a04f-041ea3dfd125 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:35.881830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c521b9b1-c850-49bf-a04f-041ea3dfd125 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:35.882098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a11d978-c9e5-4183-b877-78a707661160 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:35.882216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c521b9b1-c850-49bf-a04f-041ea3dfd125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:35.882433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a11d978-c9e5-4183-b877-78a707661160 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:35.882716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a11d978-c9e5-4183-b877-78a707661160 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:35.905465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9a11d978-c9e5-4183-b877-78a707661160 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:35.906992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9a11d978-c9e5-4183-b877-78a707661160 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:35.907340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c521b9b1-c850-49bf-a04f-041ea3dfd125 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:35.909475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c521b9b1-c850-49bf-a04f-041ea3dfd125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:35.983287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 783/1527] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:35.984209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 745/1528] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:35.992393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0f1b6-bcf2-46ce-a9aa-3820fd182da1 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:35.992682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0f1b6-bcf2-46ce-a9aa-3820fd182da1 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:35.992960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0f1b6-bcf2-46ce-a9aa-3820fd182da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:35.993613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a11d978-c9e5-4183-b877-78a707661160 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.001346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:36.009547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a11d978-c9e5-4183-b877-78a707661160 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.013499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-53d0f1b6-bcf2-46ce-a9aa-3820fd182da1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:36.015180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-53d0f1b6-bcf2-46ce-a9aa-3820fd182da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:36.018590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.019426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.019843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.020220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.020953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:36.021981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.023601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.024073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 784/1529] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:36.029170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d0f1b6-bcf2-46ce-a9aa-3820fd182da1 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.041533 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d0f1b6-bcf2-46ce-a9aa-3820fd182da1 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.047664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.048360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.048773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.049048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.049800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:36.049903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 746/1530] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:36.050199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.051849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.054439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.070094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 785/1531] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:51:36.070817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f609e-9818-49f0-82b1-69065facd10e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.071029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.077684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1b9e8c-d04c-47be-8d39-ad72e8d71306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.079672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b9e8c-d04c-47be-8d39-ad72e8d71306 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.080352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b9e8c-d04c-47be-8d39-ad72e8d71306 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.080629 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b9e8c-d04c-47be-8d39-ad72e8d71306 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.081340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1b9e8c-d04c-47be-8d39-ad72e8d71306 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.082977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64dbf6d-6ddc-4297-aa0a-bd483740996b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.083448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1b9e8c-d04c-47be-8d39-ad72e8d71306 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.104696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d987458-694b-4090-9149-82738c647754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.105319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d987458-694b-4090-9149-82738c647754 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.105605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d987458-694b-4090-9149-82738c647754 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.105879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d987458-694b-4090-9149-82738c647754 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.106528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d987458-694b-4090-9149-82738c647754 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.108478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d987458-694b-4090-9149-82738c647754 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.130334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 747/1532] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:36 2022] POST /identity/v3/projects => generated 407 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:36.133544 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1b9e8c-d04c-47be-8d39-ad72e8d71306 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.150690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 786/1533] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:36 2022] POST /identity/v3/projects => generated 369 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:36.154628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d987458-694b-4090-9149-82738c647754 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.232025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1b9e8c-d04c-47be-8d39-ad72e8d71306 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.246411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d987458-694b-4090-9149-82738c647754 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.251594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9ed725-016d-4e19-b607-dc04e6bbe27d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.252243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9ed725-016d-4e19-b607-dc04e6bbe27d admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.252618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9ed725-016d-4e19-b607-dc04e6bbe27d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.252901 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9ed725-016d-4e19-b607-dc04e6bbe27d admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.253673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9ed725-016d-4e19-b607-dc04e6bbe27d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.255897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9ed725-016d-4e19-b607-dc04e6bbe27d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.279819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 787/1534] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:36 2022] POST /identity/v3/users => generated 285 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:36.284107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9ed725-016d-4e19-b607-dc04e6bbe27d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.294129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9ed725-016d-4e19-b607-dc04e6bbe27d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.300458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c10e4f-2fa7-413c-b02f-f087efd571cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.301158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c10e4f-2fa7-413c-b02f-f087efd571cd admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.301555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c10e4f-2fa7-413c-b02f-f087efd571cd admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.301936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c10e4f-2fa7-413c-b02f-f087efd571cd admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.302705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c10e4f-2fa7-413c-b02f-f087efd571cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.303656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9343d319-b30a-4480-b957-90844ff7fcca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.304373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9343d319-b30a-4480-b957-90844ff7fcca admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.304619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c10e4f-2fa7-413c-b02f-f087efd571cd admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.304724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9343d319-b30a-4480-b957-90844ff7fcca admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.304956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9343d319-b30a-4480-b957-90844ff7fcca admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.306482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9343d319-b30a-4480-b957-90844ff7fcca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.308126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9343d319-b30a-4480-b957-90844ff7fcca admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.315849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 788/1535] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:36.320535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c10e4f-2fa7-413c-b02f-f087efd571cd admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.333295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c10e4f-2fa7-413c-b02f-f087efd571cd admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.336013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 748/1536] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:36 2022] POST /identity/v3/users => generated 304 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:36.339507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9343d319-b30a-4480-b957-90844ff7fcca admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.340563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8cf6d-d5f9-4f61-8201-0fea51077369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.341339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8cf6d-d5f9-4f61-8201-0fea51077369 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.341892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8cf6d-d5f9-4f61-8201-0fea51077369 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.342636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8cf6d-d5f9-4f61-8201-0fea51077369 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.344567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf8cf6d-d5f9-4f61-8201-0fea51077369 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.346770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf8cf6d-d5f9-4f61-8201-0fea51077369 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.350059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9343d319-b30a-4480-b957-90844ff7fcca admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.353985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:36.354845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cb90e8-6ed8-46ca-8d24-f00263698f7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.355631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cb90e8-6ed8-46ca-8d24-f00263698f7a admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.355831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cb90e8-6ed8-46ca-8d24-f00263698f7a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.356009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cb90e8-6ed8-46ca-8d24-f00263698f7a admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.356677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cb90e8-6ed8-46ca-8d24-f00263698f7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.358283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cb90e8-6ed8-46ca-8d24-f00263698f7a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.361628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 789/1537] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:36 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:36.365983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8cf6d-d5f9-4f61-8201-0fea51077369 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.375984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 749/1538] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:36.376384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8cf6d-d5f9-4f61-8201-0fea51077369 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.377060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cb90e8-6ed8-46ca-8d24-f00263698f7a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.381046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f019a6-23a0-4824-8b91-adc2f9d3cdf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.381698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f019a6-23a0-4824-8b91-adc2f9d3cdf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.381975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f019a6-23a0-4824-8b91-adc2f9d3cdf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.382251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f019a6-23a0-4824-8b91-adc2f9d3cdf1 admin admin] PATH_INFO: `/v3/projects/257c15df232146cab06af2a01271cf47/users/ba9eb845217f46159e9a41e157f7e62b/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.386831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cb90e8-6ed8-46ca-8d24-f00263698f7a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.392098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b1327-8317-4bef-ac34-12ceda295269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.392718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b1327-8317-4bef-ac34-12ceda295269 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.393003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b1327-8317-4bef-ac34-12ceda295269 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.393280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b1327-8317-4bef-ac34-12ceda295269 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.393920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0b1327-8317-4bef-ac34-12ceda295269 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.395517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0b1327-8317-4bef-ac34-12ceda295269 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.399518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f019a6-23a0-4824-8b91-adc2f9d3cdf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=257c15df232146cab06af2a01271cf47, user_id=ba9eb845217f46159e9a41e157f7e62b, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.403861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f019a6-23a0-4824-8b91-adc2f9d3cdf1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.416898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 750/1539] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:36 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:36.418535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b1327-8317-4bef-ac34-12ceda295269 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.423439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 790/1540] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:36 2022] PUT /identity/v3/projects/257c15df232146cab06af2a01271cf47/users/ba9eb845217f46159e9a41e157f7e62b/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:36.434782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b1327-8317-4bef-ac34-12ceda295269 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.442675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b53c90-9a39-4cbc-adfa-c2b0ad9a3db5 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.443132 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b53c90-9a39-4cbc-adfa-c2b0ad9a3db5 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.443573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b53c90-9a39-4cbc-adfa-c2b0ad9a3db5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.485867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c7b53c90-9a39-4cbc-adfa-c2b0ad9a3db5 None None] MFA Rules not processed for user `ba9eb845217f46159e9a41e157f7e62b`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:36.487558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c7b53c90-9a39-4cbc-adfa-c2b0ad9a3db5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:36.498823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:36.524150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.525357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.525646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.525926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] PATH_INFO: `/v3/projects/af1c8b36a2ec4d1f9090748a6aef50d3/users/a32db48b992d40ac9ae9c24584553946/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.552755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=af1c8b36a2ec4d1f9090748a6aef50d3, user_id=a32db48b992d40ac9ae9c24584553946, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.557718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.584260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 791/1542] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:36 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:36.584432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 751/1542] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:36 2022] PUT /identity/v3/projects/af1c8b36a2ec4d1f9090748a6aef50d3/users/a32db48b992d40ac9ae9c24584553946/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:36.588850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e87e7a0-f309-4840-86c0-3a87e72bf254 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.589540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e87e7a0-f309-4840-86c0-3a87e72bf254 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.589540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e87e7a0-f309-4840-86c0-3a87e72bf254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.597084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.599095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:36.630456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-3e87e7a0-f309-4840-86c0-3a87e72bf254 None None] MFA Rules not processed for user `a32db48b992d40ac9ae9c24584553946`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:36.631895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3e87e7a0-f309-4840-86c0-3a87e72bf254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:36.711485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 792/1543] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:36 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:36.712640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e87e7a0-f309-4840-86c0-3a87e72bf254 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.721351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e910c83-0171-4383-ae7d-e359ce942a9c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.725306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6ae248-14f9-4897-b0b1-2b609535bef9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a4fd11d5d85479c9aa1d87b23a13141', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.725922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6ae248-14f9-4897-b0b1-2b609535bef9 service glance] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.726267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6ae248-14f9-4897-b0b1-2b609535bef9 service glance] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.726552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6ae248-14f9-4897-b0b1-2b609535bef9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.784102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:36.791010 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e87e7a0-f309-4840-86c0-3a87e72bf254 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.796287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0758685-deba-4f97-b741-8faa9765a99b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.796904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6ae248-14f9-4897-b0b1-2b609535bef9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.797747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0758685-deba-4f97-b741-8faa9765a99b admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.798062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0758685-deba-4f97-b741-8faa9765a99b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.798391 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0758685-deba-4f97-b741-8faa9765a99b admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.798647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6ae248-14f9-4897-b0b1-2b609535bef9 service glance] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.799150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0758685-deba-4f97-b741-8faa9765a99b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.801120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0758685-deba-4f97-b741-8faa9765a99b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.809450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 752/1544] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:36 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:36.833453 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 793/1545] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:36 2022] POST /identity/v3/projects => generated 409 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:36.840944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6ae248-14f9-4897-b0b1-2b609535bef9 service glance] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.853776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6ae248-14f9-4897-b0b1-2b609535bef9 service glance] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.924824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56e015b-088a-41d0-98bc-7a9733dcdc28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.925589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e015b-088a-41d0-98bc-7a9733dcdc28 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.925944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e015b-088a-41d0-98bc-7a9733dcdc28 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.926293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e015b-088a-41d0-98bc-7a9733dcdc28 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.926998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56e015b-088a-41d0-98bc-7a9733dcdc28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.929067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56e015b-088a-41d0-98bc-7a9733dcdc28 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:36.940283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:36.962748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 753/1546] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:36 2022] POST /identity/v3/users => generated 311 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:36.966947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0758685-deba-4f97-b741-8faa9765a99b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:36.979109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0758685-deba-4f97-b741-8faa9765a99b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:36.984672 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c4ae70-81a7-4c8b-88b8-2dab3b76d417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:36.985993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c4ae70-81a7-4c8b-88b8-2dab3b76d417 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:36.986332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c4ae70-81a7-4c8b-88b8-2dab3b76d417 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:36.986547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c4ae70-81a7-4c8b-88b8-2dab3b76d417 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:36.987189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c4ae70-81a7-4c8b-88b8-2dab3b76d417 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:36.988847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c4ae70-81a7-4c8b-88b8-2dab3b76d417 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:37.004124 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 794/1547] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:37.005352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56e015b-088a-41d0-98bc-7a9733dcdc28 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:37.016337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56e015b-088a-41d0-98bc-7a9733dcdc28 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:37.021151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba69eaa-e773-4ff2-979a-8ed0c5fc4582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:37.021803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba69eaa-e773-4ff2-979a-8ed0c5fc4582 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:37.022091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba69eaa-e773-4ff2-979a-8ed0c5fc4582 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:37.022369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba69eaa-e773-4ff2-979a-8ed0c5fc4582 admin admin] PATH_INFO: `/v3/projects/f4048a52c2044ca28e3d1e0e3cbd7913/users/785da48b7f774ca594f78d5cbe0cc469/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:37.040857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba69eaa-e773-4ff2-979a-8ed0c5fc4582 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4048a52c2044ca28e3d1e0e3cbd7913, user_id=785da48b7f774ca594f78d5cbe0cc469, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:37.043899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba69eaa-e773-4ff2-979a-8ed0c5fc4582 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:37.058627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 754/1548] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:37 2022] PUT /identity/v3/projects/f4048a52c2044ca28e3d1e0e3cbd7913/users/785da48b7f774ca594f78d5cbe0cc469/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:37.063311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c4ae70-81a7-4c8b-88b8-2dab3b76d417 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:37.073984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c4ae70-81a7-4c8b-88b8-2dab3b76d417 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:37.159216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a0e3a7-77d7-4976-8b25-5b97051f7d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:37.159830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a0e3a7-77d7-4976-8b25-5b97051f7d37 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:37.160110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a0e3a7-77d7-4976-8b25-5b97051f7d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:37.160413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a0e3a7-77d7-4976-8b25-5b97051f7d37 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:37.161056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a0e3a7-77d7-4976-8b25-5b97051f7d37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:37.163777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a0e3a7-77d7-4976-8b25-5b97051f7d37 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:37.177867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 795/1549] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:37 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:37.182160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba69eaa-e773-4ff2-979a-8ed0c5fc4582 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:37.194110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba69eaa-e773-4ff2-979a-8ed0c5fc4582 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:37.199082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:37.199909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:37.200052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:37.200353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] PATH_INFO: `/v3/projects/f4048a52c2044ca28e3d1e0e3cbd7913/users/785da48b7f774ca594f78d5cbe0cc469/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:37.202898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4048a52c2044ca28e3d1e0e3cbd7913, user_id=785da48b7f774ca594f78d5cbe0cc469, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:37.205861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:37.221674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 755/1550] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:37 2022] PUT /identity/v3/projects/f4048a52c2044ca28e3d1e0e3cbd7913/users/785da48b7f774ca594f78d5cbe0cc469/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:37.232793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf02f0ad-2199-435d-a22c-14825087e017 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:37.233075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf02f0ad-2199-435d-a22c-14825087e017 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:37.233362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf02f0ad-2199-435d-a22c-14825087e017 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:37.276551 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-cf02f0ad-2199-435d-a22c-14825087e017 None None] MFA Rules not processed for user `785da48b7f774ca594f78d5cbe0cc469`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:37.278821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-cf02f0ad-2199-435d-a22c-14825087e017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:37.405306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 796/1551] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:37 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:37.418768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:37.420554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:37.479294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e469e49-ad83-4a38-b353-63c810400f5d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:37.482886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87335642-0cca-4f8a-ab9b-e65a6c469de0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:37.483483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87335642-0cca-4f8a-ab9b-e65a6c469de0 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:37.483758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87335642-0cca-4f8a-ab9b-e65a6c469de0 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:37.484879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87335642-0cca-4f8a-ab9b-e65a6c469de0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:37.494178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87335642-0cca-4f8a-ab9b-e65a6c469de0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:37.495598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87335642-0cca-4f8a-ab9b-e65a6c469de0 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:37.505868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 756/1552] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:37 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:37.679074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf02f0ad-2199-435d-a22c-14825087e017 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:37.680653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-cf02f0ad-2199-435d-a22c-14825087e017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:37.692766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf02f0ad-2199-435d-a22c-14825087e017 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:37.696288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2974b9b-d7d2-4684-ac0a-210215889a4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:37.696891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2974b9b-d7d2-4684-ac0a-210215889a4a service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:37.697168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2974b9b-d7d2-4684-ac0a-210215889a4a service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:37.697437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2974b9b-d7d2-4684-ac0a-210215889a4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:37.748096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:37.759811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2974b9b-d7d2-4684-ac0a-210215889a4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:37.761272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2974b9b-d7d2-4684-ac0a-210215889a4a service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:37.771926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 797/1553] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:37 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:39.585010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b6a928-5d22-48a7-924d-d03528c80adc None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:39.585633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b6a928-5d22-48a7-924d-d03528c80adc None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:39.585633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b6a928-5d22-48a7-924d-d03528c80adc None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:39.586920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 757/1554] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:51:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:51:39.602517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce66cab-be06-473b-a1ea-9cba0b6ee406 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:39.602517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce66cab-be06-473b-a1ea-9cba0b6ee406 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:39.603187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce66cab-be06-473b-a1ea-9cba0b6ee406 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:39.631619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6ce66cab-be06-473b-a1ea-9cba0b6ee406 None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:39.633647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6ce66cab-be06-473b-a1ea-9cba0b6ee406 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:39.748177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 798/1555] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:51:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:39.776749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b6a928-5d22-48a7-924d-d03528c80adc None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:39.779322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e6b6a928-5d22-48a7-924d-d03528c80adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:39.808819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b6a928-5d22-48a7-924d-d03528c80adc None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:39.826921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1c69a0-5b9a-48cd-b7bc-74dc850270a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:39.830728 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1c69a0-5b9a-48cd-b7bc-74dc850270a6 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:39.831364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1c69a0-5b9a-48cd-b7bc-74dc850270a6 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:39.831792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1c69a0-5b9a-48cd-b7bc-74dc850270a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:39.846873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1c69a0-5b9a-48cd-b7bc-74dc850270a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:39.848785 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1c69a0-5b9a-48cd-b7bc-74dc850270a6 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:39.863333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 758/1556] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:41.972937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce66cab-be06-473b-a1ea-9cba0b6ee406 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:41.974393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6ce66cab-be06-473b-a1ea-9cba0b6ee406 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:42.041050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce66cab-be06-473b-a1ea-9cba0b6ee406 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:42.045524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:42.046154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:42.047219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:42.047498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] PATH_INFO: `/v3/users/db638e4886ac4982912abb8ab9d317d2` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:42.048742 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=db638e4886ac4982912abb8ab9d317d2)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:42.050651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:42.095595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db638e4886ac4982912abb8ab9d317d2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.102464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db638e4886ac4982912abb8ab9d317d2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.114915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db638e4886ac4982912abb8ab9d317d2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.115875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db638e4886ac4982912abb8ab9d317d2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.116535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db638e4886ac4982912abb8ab9d317d2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.126178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:42.131675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 799/1557] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:41 2022] DELETE /identity/v3/users/db638e4886ac4982912abb8ab9d317d2 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:42.136290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a22b26-f9a5-4141-a66d-e7112cdd4bf7 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:42.136968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a22b26-f9a5-4141-a66d-e7112cdd4bf7 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:42.136968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a22b26-f9a5-4141-a66d-e7112cdd4bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:42.156889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-90a22b26-f9a5-4141-a66d-e7112cdd4bf7 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:42.158111 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-90a22b26-f9a5-4141-a66d-e7112cdd4bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:42.216993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 759/1558] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:51:42.226833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:42.228247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:42.288610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41d78cf-bd86-449b-a026-2dd0746ccc41 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:42.295167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8db2bf-aabe-4150-9f89-8e3fc5d629f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:42.296038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8db2bf-aabe-4150-9f89-8e3fc5d629f8 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:42.297086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8db2bf-aabe-4150-9f89-8e3fc5d629f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:42.297626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8db2bf-aabe-4150-9f89-8e3fc5d629f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:42.316081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8db2bf-aabe-4150-9f89-8e3fc5d629f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:42.317573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8db2bf-aabe-4150-9f89-8e3fc5d629f8 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:42.328677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 800/1559] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:42.653792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a22b26-f9a5-4141-a66d-e7112cdd4bf7 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:42.655256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-90a22b26-f9a5-4141-a66d-e7112cdd4bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:42.668099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a22b26-f9a5-4141-a66d-e7112cdd4bf7 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:42.674078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:42.674749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:42.675030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:42.675304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] PATH_INFO: `/v3/projects/3c744e130db340d489e8d60444f9490e` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:42.676564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c744e130db340d489e8d60444f9490e)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:42.678513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:42.718870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:INSERT=4,keystone:SELECT=93,keystone:DELETE=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:42.727819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c744e130db340d489e8d60444f9490e', 'request_id': 'req-01f44fc2-7b7c-410c-a487-1011f2ca3006'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.734912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c744e130db340d489e8d60444f9490e', 'request_id': 'req-01f44fc2-7b7c-410c-a487-1011f2ca3006'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.741827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c744e130db340d489e8d60444f9490e', 'request_id': 'req-01f44fc2-7b7c-410c-a487-1011f2ca3006'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.756971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] The token cache is being invalidate because project 3c744e130db340d489e8d60444f9490e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:42.757140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.757807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.759307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 760/1560] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:42 2022] DELETE /identity/v3/projects/3c744e130db340d489e8d60444f9490e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:42.763504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8db2bf-aabe-4150-9f89-8e3fc5d629f8 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:42.774080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8db2bf-aabe-4150-9f89-8e3fc5d629f8 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:42.838992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:42.839635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:42.839916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:42.840238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] PATH_INFO: `/v3/users/69a54f1f4eca49ffaaf29313abdbcfd8` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:42.841440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] RBAC: Authorizing `identity:delete_user(user_id=69a54f1f4eca49ffaaf29313abdbcfd8)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:42.843928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:42.895733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '69a54f1f4eca49ffaaf29313abdbcfd8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.903169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '69a54f1f4eca49ffaaf29313abdbcfd8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.916952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '69a54f1f4eca49ffaaf29313abdbcfd8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.917918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '69a54f1f4eca49ffaaf29313abdbcfd8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.918532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-69f2412a-4fe4-4572-a6f7-0e6f13c11a17 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '69a54f1f4eca49ffaaf29313abdbcfd8'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:42.934095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 801/1561] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:42 2022] DELETE /identity/v3/users/69a54f1f4eca49ffaaf29313abdbcfd8 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:43.250705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:43.252212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:43.268656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f44fc2-7b7c-410c-a487-1011f2ca3006 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:43.326786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:43.327419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.327805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.328296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] PATH_INFO: `/v3/projects/37e61ff111a941e78e57281f16baed64` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.329202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37e61ff111a941e78e57281f16baed64)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:43.331152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:43.377816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37e61ff111a941e78e57281f16baed64', 'request_id': 'req-cb7f86fd-668b-42f7-b7d1-d98d127ca552'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:43.383258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37e61ff111a941e78e57281f16baed64', 'request_id': 'req-cb7f86fd-668b-42f7-b7d1-d98d127ca552'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:43.389460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37e61ff111a941e78e57281f16baed64', 'request_id': 'req-cb7f86fd-668b-42f7-b7d1-d98d127ca552'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:43.403082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] The token cache is being invalidate because project 37e61ff111a941e78e57281f16baed64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:43.403428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:43.404093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cb7f86fd-668b-42f7-b7d1-d98d127ca552 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:43.406866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 761/1562] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:43 2022] DELETE /identity/v3/projects/37e61ff111a941e78e57281f16baed64 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:43.411774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ff4d3a-2547-4754-a5a3-a316dd2d0611 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.412372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ff4d3a-2547-4754-a5a3-a316dd2d0611 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.412372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ff4d3a-2547-4754-a5a3-a316dd2d0611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.432851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-16ff4d3a-2547-4754-a5a3-a316dd2d0611 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:43.434049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-16ff4d3a-2547-4754-a5a3-a316dd2d0611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:43.451149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:43.497706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 802/1563] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:43.505570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564cec44-eb22-4862-b903-7b01df41a2a4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.505857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564cec44-eb22-4862-b903-7b01df41a2a4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.506133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564cec44-eb22-4862-b903-7b01df41a2a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.527333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-564cec44-eb22-4862-b903-7b01df41a2a4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:43.528935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-564cec44-eb22-4862-b903-7b01df41a2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:43.538151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 762/1564] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:43.545213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ff4d3a-2547-4754-a5a3-a316dd2d0611 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:43.555794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ff4d3a-2547-4754-a5a3-a316dd2d0611 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:43.560437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:43.561153 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.561441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.561719 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.562936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:43.563246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:43.564885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:43.577294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 803/1565] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:43.581883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564cec44-eb22-4862-b903-7b01df41a2a4 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:43.591489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564cec44-eb22-4862-b903-7b01df41a2a4 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:43.597005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ead42f-0c5b-426d-918e-9571143a5ae3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:43.597624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ead42f-0c5b-426d-918e-9571143a5ae3 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.597933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ead42f-0c5b-426d-918e-9571143a5ae3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.598209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ead42f-0c5b-426d-918e-9571143a5ae3 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.598858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ead42f-0c5b-426d-918e-9571143a5ae3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:43.600874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ead42f-0c5b-426d-918e-9571143a5ae3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:43.632908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 763/1566] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:43 2022] POST /identity/v3/projects => generated 387 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:43.634416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:43.645001 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633983c4-839e-4f86-b128-15c75eb7fb16 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:43.711973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd37d19-b684-4f6b-b33c-93f6c8cfe60e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:43.712745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd37d19-b684-4f6b-b33c-93f6c8cfe60e admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.713339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd37d19-b684-4f6b-b33c-93f6c8cfe60e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.713591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd37d19-b684-4f6b-b33c-93f6c8cfe60e admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.715054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd37d19-b684-4f6b-b33c-93f6c8cfe60e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:43.717050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd37d19-b684-4f6b-b33c-93f6c8cfe60e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:43.769647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 804/1567] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:43 2022] POST /identity/v3/users => generated 294 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:43.771085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ead42f-0c5b-426d-918e-9571143a5ae3 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:43.782140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ead42f-0c5b-426d-918e-9571143a5ae3 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:43.787747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb16f6d-09b8-4bf3-a947-fa0b1caa682c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:43.788407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb16f6d-09b8-4bf3-a947-fa0b1caa682c admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.788687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb16f6d-09b8-4bf3-a947-fa0b1caa682c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.788958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb16f6d-09b8-4bf3-a947-fa0b1caa682c admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.789604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb16f6d-09b8-4bf3-a947-fa0b1caa682c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:43.792147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb16f6d-09b8-4bf3-a947-fa0b1caa682c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:43.809411 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 764/1568] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:43 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:43.814178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd37d19-b684-4f6b-b33c-93f6c8cfe60e admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:43.825211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd37d19-b684-4f6b-b33c-93f6c8cfe60e admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:43.831349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a501578d-f17c-4fd7-87ea-060d44bb2a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:43.831950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a501578d-f17c-4fd7-87ea-060d44bb2a62 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.832834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a501578d-f17c-4fd7-87ea-060d44bb2a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.833113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a501578d-f17c-4fd7-87ea-060d44bb2a62 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.833853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a501578d-f17c-4fd7-87ea-060d44bb2a62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:43.835442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a501578d-f17c-4fd7-87ea-060d44bb2a62 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:43.848169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 805/1569] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:43 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:43.852251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb16f6d-09b8-4bf3-a947-fa0b1caa682c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:43.863287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb16f6d-09b8-4bf3-a947-fa0b1caa682c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:43.868184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4b2c8e-cd86-4829-85bd-b1aa711f5db9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:43.868867 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4b2c8e-cd86-4829-85bd-b1aa711f5db9 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.869171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4b2c8e-cd86-4829-85bd-b1aa711f5db9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.869444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4b2c8e-cd86-4829-85bd-b1aa711f5db9 admin admin] PATH_INFO: `/v3/projects/46451dd7f7114302ac9cf1a9a46cb8c9/users/e2d5a0ce23f94f84b7a26e2eae7f1922/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.884668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4b2c8e-cd86-4829-85bd-b1aa711f5db9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=46451dd7f7114302ac9cf1a9a46cb8c9, user_id=e2d5a0ce23f94f84b7a26e2eae7f1922, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:43.887395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4b2c8e-cd86-4829-85bd-b1aa711f5db9 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:43.889592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:43.903558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 765/1570] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:43 2022] PUT /identity/v3/projects/46451dd7f7114302ac9cf1a9a46cb8c9/users/e2d5a0ce23f94f84b7a26e2eae7f1922/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:43.915601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65204e6b-2edb-43f6-bfcf-7554807b0fdd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:43.915886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65204e6b-2edb-43f6-bfcf-7554807b0fdd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:43.916188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65204e6b-2edb-43f6-bfcf-7554807b0fdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:43.945380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-65204e6b-2edb-43f6-bfcf-7554807b0fdd None None] MFA Rules not processed for user `e2d5a0ce23f94f84b7a26e2eae7f1922`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:43.946720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-65204e6b-2edb-43f6-bfcf-7554807b0fdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:44.012856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 806/1571] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:43 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:44.018010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4b2c8e-cd86-4829-85bd-b1aa711f5db9 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.080108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4b2c8e-cd86-4829-85bd-b1aa711f5db9 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.084703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bf4ea1-2926-4c5b-bcac-8642982c7783 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.085300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf4ea1-2926-4c5b-bcac-8642982c7783 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.085581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf4ea1-2926-4c5b-bcac-8642982c7783 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.085853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf4ea1-2926-4c5b-bcac-8642982c7783 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.086489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bf4ea1-2926-4c5b-bcac-8642982c7783 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.088419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bf4ea1-2926-4c5b-bcac-8642982c7783 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.116156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 766/1572] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:44 2022] POST /identity/v3/projects => generated 387 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:44.120305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65204e6b-2edb-43f6-bfcf-7554807b0fdd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.131046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65204e6b-2edb-43f6-bfcf-7554807b0fdd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.142968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:44.187743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb8183-7da6-44c2-af2e-4682b3b64019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.188395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb8183-7da6-44c2-af2e-4682b3b64019 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.188674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb8183-7da6-44c2-af2e-4682b3b64019 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.188949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb8183-7da6-44c2-af2e-4682b3b64019 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.189573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cb8183-7da6-44c2-af2e-4682b3b64019 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.191477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cb8183-7da6-44c2-af2e-4682b3b64019 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.214398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 807/1573] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:44 2022] POST /identity/v3/users => generated 294 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:44.219229 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bf4ea1-2926-4c5b-bcac-8642982c7783 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.229622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bf4ea1-2926-4c5b-bcac-8642982c7783 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.235456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eecee4-f898-48b6-a835-49e9928b0f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.236174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eecee4-f898-48b6-a835-49e9928b0f56 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.236460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eecee4-f898-48b6-a835-49e9928b0f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.236739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eecee4-f898-48b6-a835-49e9928b0f56 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.237388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77eecee4-f898-48b6-a835-49e9928b0f56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.238984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77eecee4-f898-48b6-a835-49e9928b0f56 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.250327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 767/1574] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:44.254576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb8183-7da6-44c2-af2e-4682b3b64019 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.264816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb8183-7da6-44c2-af2e-4682b3b64019 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.270323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6386a2c-1115-4b79-8db5-378931919ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.270916 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6386a2c-1115-4b79-8db5-378931919ab6 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.271190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6386a2c-1115-4b79-8db5-378931919ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.271582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6386a2c-1115-4b79-8db5-378931919ab6 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.272331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6386a2c-1115-4b79-8db5-378931919ab6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.273947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6386a2c-1115-4b79-8db5-378931919ab6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.288503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 808/1575] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:44.289692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eecee4-f898-48b6-a835-49e9928b0f56 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.300428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eecee4-f898-48b6-a835-49e9928b0f56 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.305295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7d11b-3e42-464e-a02d-24c4785e6900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.305950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7d11b-3e42-464e-a02d-24c4785e6900 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.306233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7d11b-3e42-464e-a02d-24c4785e6900 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.306505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7d11b-3e42-464e-a02d-24c4785e6900 admin admin] PATH_INFO: `/v3/projects/ae13cea6133c40878b57aaf15ca0e8f4/users/9f7aaa388a4b4014aca92d036375013f/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.322330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a7d11b-3e42-464e-a02d-24c4785e6900 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae13cea6133c40878b57aaf15ca0e8f4, user_id=9f7aaa388a4b4014aca92d036375013f, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.325057 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a7d11b-3e42-464e-a02d-24c4785e6900 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.339400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 768/1576] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:44 2022] PUT /identity/v3/projects/ae13cea6133c40878b57aaf15ca0e8f4/users/9f7aaa388a4b4014aca92d036375013f/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:44.350705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df64ff-ef62-467c-8570-90e846dadcac None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.351056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df64ff-ef62-467c-8570-90e846dadcac None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.351411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df64ff-ef62-467c-8570-90e846dadcac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.379954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c7df64ff-ef62-467c-8570-90e846dadcac None None] MFA Rules not processed for user `9f7aaa388a4b4014aca92d036375013f`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:44.381180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c7df64ff-ef62-467c-8570-90e846dadcac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:44.445770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 809/1577] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:44 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:44.450417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7d11b-3e42-464e-a02d-24c4785e6900 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.513974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7d11b-3e42-464e-a02d-24c4785e6900 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.519572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd58491-7fe8-4aaa-ad30-dffc218db2eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.520213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd58491-7fe8-4aaa-ad30-dffc218db2eb admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.520501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd58491-7fe8-4aaa-ad30-dffc218db2eb admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.520777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd58491-7fe8-4aaa-ad30-dffc218db2eb admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.521423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd58491-7fe8-4aaa-ad30-dffc218db2eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.523710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd58491-7fe8-4aaa-ad30-dffc218db2eb admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.553430 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 769/1578] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:44 2022] POST /identity/v3/projects => generated 383 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:44.564033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7df64ff-ef62-467c-8570-90e846dadcac None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.574732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7df64ff-ef62-467c-8570-90e846dadcac None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.632483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe6cb9d-0b8b-4374-82e3-b2cdad08a90e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.633083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe6cb9d-0b8b-4374-82e3-b2cdad08a90e admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.633359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe6cb9d-0b8b-4374-82e3-b2cdad08a90e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.633627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe6cb9d-0b8b-4374-82e3-b2cdad08a90e admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.634279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe6cb9d-0b8b-4374-82e3-b2cdad08a90e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.636195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe6cb9d-0b8b-4374-82e3-b2cdad08a90e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.658854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 810/1579] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:44 2022] POST /identity/v3/users => generated 298 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:44.662876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd58491-7fe8-4aaa-ad30-dffc218db2eb admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.673165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd58491-7fe8-4aaa-ad30-dffc218db2eb admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.677905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55e2840-0bea-4acf-a209-97b278ba2a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.678527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e2840-0bea-4acf-a209-97b278ba2a07 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.678807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e2840-0bea-4acf-a209-97b278ba2a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.679078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e2840-0bea-4acf-a209-97b278ba2a07 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.679716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55e2840-0bea-4acf-a209-97b278ba2a07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.681325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55e2840-0bea-4acf-a209-97b278ba2a07 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.692042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 770/1580] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:44.696302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe6cb9d-0b8b-4374-82e3-b2cdad08a90e admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.707513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe6cb9d-0b8b-4374-82e3-b2cdad08a90e admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.712111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a14579e-4f41-4431-8e52-c7eca4e76fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.712807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a14579e-4f41-4431-8e52-c7eca4e76fda admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.713083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a14579e-4f41-4431-8e52-c7eca4e76fda admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.713421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a14579e-4f41-4431-8e52-c7eca4e76fda admin admin] PATH_INFO: `/v3/projects/70e3997385ec481a9052a61d1157bbc6/users/b7affd8d7752418895bf0b7ea5036232/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.727195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:44.730908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a14579e-4f41-4431-8e52-c7eca4e76fda admin admin] RBAC: Authorizing `identity:create_grant(project_id=70e3997385ec481a9052a61d1157bbc6, user_id=b7affd8d7752418895bf0b7ea5036232, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.734653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a14579e-4f41-4431-8e52-c7eca4e76fda admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.750245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 811/1581] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:44 2022] PUT /identity/v3/projects/70e3997385ec481a9052a61d1157bbc6/users/b7affd8d7752418895bf0b7ea5036232/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:44.754165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55e2840-0bea-4acf-a209-97b278ba2a07 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.767270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55e2840-0bea-4acf-a209-97b278ba2a07 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.783608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:44.835430 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6de915-7288-43c9-9f7e-f522bfea6766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.836038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6de915-7288-43c9-9f7e-f522bfea6766 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.836670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6de915-7288-43c9-9f7e-f522bfea6766 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.836954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6de915-7288-43c9-9f7e-f522bfea6766 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.837599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6de915-7288-43c9-9f7e-f522bfea6766 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.839202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6de915-7288-43c9-9f7e-f522bfea6766 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.853235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 771/1582] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:44 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:44.854316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a14579e-4f41-4431-8e52-c7eca4e76fda admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:44.866345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a14579e-4f41-4431-8e52-c7eca4e76fda admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:44.871410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:44.872371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.872825 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.874009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] PATH_INFO: `/v3/projects/70e3997385ec481a9052a61d1157bbc6/users/b7affd8d7752418895bf0b7ea5036232/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.875902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] RBAC: Authorizing `identity:create_grant(project_id=70e3997385ec481a9052a61d1157bbc6, user_id=b7affd8d7752418895bf0b7ea5036232, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:44.878604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:44.893344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 812/1583] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:44 2022] PUT /identity/v3/projects/70e3997385ec481a9052a61d1157bbc6/users/b7affd8d7752418895bf0b7ea5036232/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:44.905729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb7e65-9aa9-4b0c-aef9-3bf55620fcfa None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:44.906016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb7e65-9aa9-4b0c-aef9-3bf55620fcfa None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:44.906292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb7e65-9aa9-4b0c-aef9-3bf55620fcfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:44.955293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-86eb7e65-9aa9-4b0c-aef9-3bf55620fcfa None None] MFA Rules not processed for user `b7affd8d7752418895bf0b7ea5036232`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:44.956558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-86eb7e65-9aa9-4b0c-aef9-3bf55620fcfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:45.040570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 772/1584] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:44 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:45.052032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:45.053826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:45.120109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c50ce7-d302-49ac-845a-d41bd21f270c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:45.123735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:45.124465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:45.124904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:45.125126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:45.134469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:45.136021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:45.146020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 813/1585] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:45 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:45.717812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb7e65-9aa9-4b0c-aef9-3bf55620fcfa None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:45.727838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb7e65-9aa9-4b0c-aef9-3bf55620fcfa None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:45.731500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5888225b-302f-45c6-b11d-193a1a8939bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:45.732098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888225b-302f-45c6-b11d-193a1a8939bb service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:45.732427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888225b-302f-45c6-b11d-193a1a8939bb service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:45.732701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888225b-302f-45c6-b11d-193a1a8939bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:45.792923 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5888225b-302f-45c6-b11d-193a1a8939bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:45.794270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5888225b-302f-45c6-b11d-193a1a8939bb service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:45.803446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 773/1586] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:45 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:54.250440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:54.251893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:54.324736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32609b08-6c75-41c3-8170-17ce592e27b2 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:54.330169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:54.330819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:54.331101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:54.331374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] PATH_INFO: `/v3/users/a32db48b992d40ac9ae9c24584553946` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:54.332562 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a32db48b992d40ac9ae9c24584553946)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:54.334493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:54.402093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a32db48b992d40ac9ae9c24584553946'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:54.413738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a32db48b992d40ac9ae9c24584553946'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:54.431967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a32db48b992d40ac9ae9c24584553946'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:54.433172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a32db48b992d40ac9ae9c24584553946'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:54.433915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a32db48b992d40ac9ae9c24584553946'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:54.450400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:54.460014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 814/1587] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:54 2022] DELETE /identity/v3/users/a32db48b992d40ac9ae9c24584553946 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:54.463429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d9944-a903-4626-be31-8ddb5d7bea9c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:54.464574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d9944-a903-4626-be31-8ddb5d7bea9c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:54.465292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d9944-a903-4626-be31-8ddb5d7bea9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:54.535939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-115d9944-a903-4626-be31-8ddb5d7bea9c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:54.535939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-115d9944-a903-4626-be31-8ddb5d7bea9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:54.616981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:54.664006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 774/1588] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:54.730581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:54.732421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:54.821682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f66452c-5de5-4b3a-8083-6a6fb9906546 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:54.827963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:54.829178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:54.829887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:54.830233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:54.840706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:54.843696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:54.861808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 815/1589] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:55.567319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115d9944-a903-4626-be31-8ddb5d7bea9c None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:55.572586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-115d9944-a903-4626-be31-8ddb5d7bea9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:55.691783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115d9944-a903-4626-be31-8ddb5d7bea9c None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:55.700286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:55.701161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:55.701696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:55.702269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:55.706879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:55.807488 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:55.809447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:55.824708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 775/1590] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:55 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:55.837874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:55.839465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:55.858135 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f0a43e-3c5b-492a-b6a2-13d7216dcfcc service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:55.865138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145343c3-9f80-420a-af62-10336849336c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:55.866182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145343c3-9f80-420a-af62-10336849336c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:55.866525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145343c3-9f80-420a-af62-10336849336c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:55.866902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145343c3-9f80-420a-af62-10336849336c admin admin] PATH_INFO: `/v3/projects/af1c8b36a2ec4d1f9090748a6aef50d3` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:55.868077 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145343c3-9f80-420a-af62-10336849336c admin admin] RBAC: Authorizing `identity:delete_project(project_id=af1c8b36a2ec4d1f9090748a6aef50d3)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:55.870309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145343c3-9f80-420a-af62-10336849336c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:55.923958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-145343c3-9f80-420a-af62-10336849336c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af1c8b36a2ec4d1f9090748a6aef50d3', 'request_id': 'req-145343c3-9f80-420a-af62-10336849336c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:55.935644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-145343c3-9f80-420a-af62-10336849336c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af1c8b36a2ec4d1f9090748a6aef50d3', 'request_id': 'req-145343c3-9f80-420a-af62-10336849336c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:55.946592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-145343c3-9f80-420a-af62-10336849336c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af1c8b36a2ec4d1f9090748a6aef50d3', 'request_id': 'req-145343c3-9f80-420a-af62-10336849336c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:55.971818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-145343c3-9f80-420a-af62-10336849336c admin admin] The token cache is being invalidate because project af1c8b36a2ec4d1f9090748a6aef50d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:55.972870 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-145343c3-9f80-420a-af62-10336849336c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:55.982922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-145343c3-9f80-420a-af62-10336849336c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:55.986338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 816/1591] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:55 2022] DELETE /identity/v3/projects/af1c8b36a2ec4d1f9090748a6aef50d3 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:55.996645 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:56.010708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdab5e15-6c6d-4fe1-a0fb-2f7c5dc2df5e service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:56.087533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:56.088313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:56.088727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:56.089071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] PATH_INFO: `/v3/users/785da48b7f774ca594f78d5cbe0cc469` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:56.090312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=785da48b7f774ca594f78d5cbe0cc469)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:56.092259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:56.147554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:56.160403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '785da48b7f774ca594f78d5cbe0cc469'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:56.172461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '785da48b7f774ca594f78d5cbe0cc469'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:56.191150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '785da48b7f774ca594f78d5cbe0cc469'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:56.193694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '785da48b7f774ca594f78d5cbe0cc469'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:56.194679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5a21aebf-7f16-4f21-8b63-79601532add8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '785da48b7f774ca594f78d5cbe0cc469'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:56.228483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 776/1592] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:55 2022] DELETE /identity/v3/users/785da48b7f774ca594f78d5cbe0cc469 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:56.643487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145343c3-9f80-420a-af62-10336849336c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:56.645000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-145343c3-9f80-420a-af62-10336849336c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:56.674855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145343c3-9f80-420a-af62-10336849336c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:56.813745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:56.814360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:56.814644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:56.814990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] PATH_INFO: `/v3/users/03e43b45b5d74ff79f28395b986719f0` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:56.816914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=03e43b45b5d74ff79f28395b986719f0)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:56.819836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:56.872537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:56.942784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03e43b45b5d74ff79f28395b986719f0'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:56.965397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03e43b45b5d74ff79f28395b986719f0'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.003277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03e43b45b5d74ff79f28395b986719f0'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.004558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03e43b45b5d74ff79f28395b986719f0'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.005974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03e43b45b5d74ff79f28395b986719f0'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.028292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 817/1593] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:56 2022] DELETE /identity/v3/users/03e43b45b5d74ff79f28395b986719f0 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:57.034322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de19d7-24d1-4af0-a9d2-e81a831fbb1b None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.035501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de19d7-24d1-4af0-a9d2-e81a831fbb1b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.035968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de19d7-24d1-4af0-a9d2-e81a831fbb1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.075174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-14de19d7-24d1-4af0-a9d2-e81a831fbb1b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:57.076651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-14de19d7-24d1-4af0-a9d2-e81a831fbb1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:57.162410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 777/1594] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:57.178282 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:57.179846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:57.245746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6447da1c-96e4-4a58-ad5b-50f5c77063bf admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:57.250966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4bf2da-ee89-463e-a40f-c12c86f5db0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:57.251672 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4bf2da-ee89-463e-a40f-c12c86f5db0b service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.252045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4bf2da-ee89-463e-a40f-c12c86f5db0b service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.252420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4bf2da-ee89-463e-a40f-c12c86f5db0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.265505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4bf2da-ee89-463e-a40f-c12c86f5db0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:57.267248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4bf2da-ee89-463e-a40f-c12c86f5db0b service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:57.280486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 818/1595] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:51:57.376055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de19d7-24d1-4af0-a9d2-e81a831fbb1b None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:57.378735 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-14de19d7-24d1-4af0-a9d2-e81a831fbb1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:57.404428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de19d7-24d1-4af0-a9d2-e81a831fbb1b None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:57.410571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:57.411212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.411494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.411761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] PATH_INFO: `/v3/projects/f4048a52c2044ca28e3d1e0e3cbd7913` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.412960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4048a52c2044ca28e3d1e0e3cbd7913)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:57.414973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:57.490230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4048a52c2044ca28e3d1e0e3cbd7913', 'request_id': 'req-94847725-2dde-42c4-a88c-49fd32b9381d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.496854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4048a52c2044ca28e3d1e0e3cbd7913', 'request_id': 'req-94847725-2dde-42c4-a88c-49fd32b9381d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.504376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4048a52c2044ca28e3d1e0e3cbd7913', 'request_id': 'req-94847725-2dde-42c4-a88c-49fd32b9381d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.525299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] The token cache is being invalidate because project f4048a52c2044ca28e3d1e0e3cbd7913 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:57.525299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.525299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-94847725-2dde-42c4-a88c-49fd32b9381d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.530122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 778/1596] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:57 2022] DELETE /identity/v3/projects/f4048a52c2044ca28e3d1e0e3cbd7913 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:57.533217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c7611-b911-46fd-979e-605875e243c6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.533404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c7611-b911-46fd-979e-605875e243c6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.533706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c7611-b911-46fd-979e-605875e243c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.555058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a43c7611-b911-46fd-979e-605875e243c6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:57.556309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a43c7611-b911-46fd-979e-605875e243c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:57.628632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 819/1597] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:57.637819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2bec33-fb28-4cba-8052-3ac16b779efa None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.638196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2bec33-fb28-4cba-8052-3ac16b779efa None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.638480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2bec33-fb28-4cba-8052-3ac16b779efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.663700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7d2bec33-fb28-4cba-8052-3ac16b779efa None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:57.665279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7d2bec33-fb28-4cba-8052-3ac16b779efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:57.677603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43c7611-b911-46fd-979e-605875e243c6 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:57.678901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 779/1598] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:57.679431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a43c7611-b911-46fd-979e-605875e243c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:57.683015 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2bec33-fb28-4cba-8052-3ac16b779efa None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:57.695603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2bec33-fb28-4cba-8052-3ac16b779efa None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:57.700907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:57.701452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43c7611-b911-46fd-979e-605875e243c6 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:57.703630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.704013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.704407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.705181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:57.705573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:57.707638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:57.708354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:57.708991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.717204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.718311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] PATH_INFO: `/v3/projects/2488ac7bcf98445a8a25f81c38e18afe` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.720863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2488ac7bcf98445a8a25f81c38e18afe)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:57.722770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:57.725649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 780/1599] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:57.729998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:57.747851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e74bf34-422c-4945-a4c4-9b772ea5cbc2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:57.764649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d537d46-3cfd-4ffb-9071-418c991dcce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:57.765625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d537d46-3cfd-4ffb-9071-418c991dcce3 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.765927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d537d46-3cfd-4ffb-9071-418c991dcce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.766213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d537d46-3cfd-4ffb-9071-418c991dcce3 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.767110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d537d46-3cfd-4ffb-9071-418c991dcce3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:57.770189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d537d46-3cfd-4ffb-9071-418c991dcce3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:57.789306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:57.790622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:57.797919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2488ac7bcf98445a8a25f81c38e18afe', 'request_id': 'req-85bc3e81-af8b-4e90-86c9-992c66a68539'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.807041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 781/1600] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:57 2022] POST /identity/v3/projects => generated 379 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:57.810799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2488ac7bcf98445a8a25f81c38e18afe', 'request_id': 'req-85bc3e81-af8b-4e90-86c9-992c66a68539'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.814176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d537d46-3cfd-4ffb-9071-418c991dcce3 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:57.821782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2488ac7bcf98445a8a25f81c38e18afe', 'request_id': 'req-85bc3e81-af8b-4e90-86c9-992c66a68539'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.838667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] The token cache is being invalidate because project 2488ac7bcf98445a8a25f81c38e18afe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:57.839122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.840038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:57.844975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 820/1601] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:57 2022] DELETE /identity/v3/projects/2488ac7bcf98445a8a25f81c38e18afe => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:51:57.845941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:57.849746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:57.939399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bc3e81-af8b-4e90-86c9-992c66a68539 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:57.950633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d537d46-3cfd-4ffb-9071-418c991dcce3 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:57.955705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:57.956432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:57.956786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:57.957128 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:57.957831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:57.959883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:57.999310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 782/1602] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:57 2022] POST /identity/v3/users => generated 290 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:58.003303 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:58.005358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:58.013262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.014038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.014325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.016209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] PATH_INFO: `/v3/users/b98ba8906d524e099f6f4dd1871871ea` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.016628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b98ba8906d524e099f6f4dd1871871ea)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.018545 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.023043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed51022f-1059-45fc-8d2e-0c4a6da613c6 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:58.030188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d919b41-7ec8-4d3c-bd98-174a43f258b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.032959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d919b41-7ec8-4d3c-bd98-174a43f258b4 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.033522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d919b41-7ec8-4d3c-bd98-174a43f258b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.034072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d919b41-7ec8-4d3c-bd98-174a43f258b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.035243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d919b41-7ec8-4d3c-bd98-174a43f258b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.037133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d919b41-7ec8-4d3c-bd98-174a43f258b4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.063322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 783/1603] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:58 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:58.066239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d919b41-7ec8-4d3c-bd98-174a43f258b4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:58.086509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b98ba8906d524e099f6f4dd1871871ea'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:58.087217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d919b41-7ec8-4d3c-bd98-174a43f258b4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:58.094970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b98ba8906d524e099f6f4dd1871871ea'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:58.099270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.099992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.100397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.100765 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.102829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.103195 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.117975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b98ba8906d524e099f6f4dd1871871ea'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:58.118739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b98ba8906d524e099f6f4dd1871871ea'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:58.119376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c8afd892-73a1-44c7-8579-b4a67a7100e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b98ba8906d524e099f6f4dd1871871ea'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:58.124346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 784/1604] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:58 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:58.129243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:58.131897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:58.133854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:58.141420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 821/1605] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:57 2022] DELETE /identity/v3/users/b98ba8906d524e099f6f4dd1871871ea => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:58.151888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf7cb21-4adf-4959-a6db-368b845a6cc1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:58.277861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f021191-ccca-4699-8d86-de03e24985df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.278887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f021191-ccca-4699-8d86-de03e24985df admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.279313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f021191-ccca-4699-8d86-de03e24985df admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.279746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f021191-ccca-4699-8d86-de03e24985df admin admin] PATH_INFO: `/v3/projects/150c629df022430da59901564c3e63bf/users/40fb3a560b5e49f1940f77a357eaaf3b/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.300215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f021191-ccca-4699-8d86-de03e24985df admin admin] RBAC: Authorizing `identity:create_grant(project_id=150c629df022430da59901564c3e63bf, user_id=40fb3a560b5e49f1940f77a357eaaf3b, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.303114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f021191-ccca-4699-8d86-de03e24985df admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.325355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 785/1606] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:58 2022] PUT /identity/v3/projects/150c629df022430da59901564c3e63bf/users/40fb3a560b5e49f1940f77a357eaaf3b/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:58.336856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a80465-f13a-4195-8d23-314e3f05ed1c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.337366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a80465-f13a-4195-8d23-314e3f05ed1c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.337965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a80465-f13a-4195-8d23-314e3f05ed1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.407904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-60a80465-f13a-4195-8d23-314e3f05ed1c None None] MFA Rules not processed for user `40fb3a560b5e49f1940f77a357eaaf3b`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:58.409333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-60a80465-f13a-4195-8d23-314e3f05ed1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:58.534911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 822/1607] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:58.539635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f021191-ccca-4699-8d86-de03e24985df admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:58.592339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:58.614705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f021191-ccca-4699-8d86-de03e24985df admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:58.620957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.621574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.621852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.622122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.622771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.625897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.668719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 786/1608] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:58 2022] POST /identity/v3/projects => generated 379 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:58.672953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a80465-f13a-4195-8d23-314e3f05ed1c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:58.694058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a80465-f13a-4195-8d23-314e3f05ed1c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:58.782462 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6558c38-64af-47b7-8cbb-8b782c06a5f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.784858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6558c38-64af-47b7-8cbb-8b782c06a5f7 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.785594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6558c38-64af-47b7-8cbb-8b782c06a5f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.786628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6558c38-64af-47b7-8cbb-8b782c06a5f7 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.787577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6558c38-64af-47b7-8cbb-8b782c06a5f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.790184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6558c38-64af-47b7-8cbb-8b782c06a5f7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.820705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 823/1609] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:58 2022] POST /identity/v3/users => generated 290 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:58.821751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:58.823893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:58.824097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6558c38-64af-47b7-8cbb-8b782c06a5f7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:58.835439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6558c38-64af-47b7-8cbb-8b782c06a5f7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:58.842171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.843029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.843179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.843452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.844174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.844729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1cdaae-b40a-448a-a0e8-2abca14d5e40 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:58.845794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.853027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.854348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.854920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.855747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] PATH_INFO: `/v3/projects/9ba8c5dfb3d04ff098905bd7841ad64b` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.858945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ba8c5dfb3d04ff098905bd7841ad64b)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.861809 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.877417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 824/1610] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:58 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:58.878795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:58.881104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:58.906230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a574c2-3059-4a8e-8d14-0379f54d6d92 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:58.917817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:58.920020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:58.920428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:58.921587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] PATH_INFO: `/v3/users/e2d5a0ce23f94f84b7a26e2eae7f1922` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:58.926481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2d5a0ce23f94f84b7a26e2eae7f1922)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:58.930685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:58.949668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ba8c5dfb3d04ff098905bd7841ad64b', 'request_id': 'req-e184960d-c930-4ef2-99ea-2f248c251496'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:58.963873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ba8c5dfb3d04ff098905bd7841ad64b', 'request_id': 'req-e184960d-c930-4ef2-99ea-2f248c251496'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:58.983774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ba8c5dfb3d04ff098905bd7841ad64b', 'request_id': 'req-e184960d-c930-4ef2-99ea-2f248c251496'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:59.011863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] The token cache is being invalidate because project 9ba8c5dfb3d04ff098905bd7841ad64b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:51:59.012892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:59.013788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:59.016108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 787/1611] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:51:58 2022] DELETE /identity/v3/projects/9ba8c5dfb3d04ff098905bd7841ad64b => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:59.016520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2d5a0ce23f94f84b7a26e2eae7f1922'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:59.017264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.018838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:59.026399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2d5a0ce23f94f84b7a26e2eae7f1922'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:59.031743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:59.042679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2d5a0ce23f94f84b7a26e2eae7f1922'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:59.044381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2d5a0ce23f94f84b7a26e2eae7f1922'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:59.045198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a32c9f86-2e29-4911-8c90-ba7e45dfc84c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2d5a0ce23f94f84b7a26e2eae7f1922'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:51:59.062563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 825/1612] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:51:58 2022] DELETE /identity/v3/users/e2d5a0ce23f94f84b7a26e2eae7f1922 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:59.071284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46097012-5ac9-4ac3-ae35-9e75e7461031 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.071284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46097012-5ac9-4ac3-ae35-9e75e7461031 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.071284 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46097012-5ac9-4ac3-ae35-9e75e7461031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.090682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-46097012-5ac9-4ac3-ae35-9e75e7461031 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:59.091874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-46097012-5ac9-4ac3-ae35-9e75e7461031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:59.095784 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184960d-c930-4ef2-99ea-2f248c251496 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.160447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 826/1613] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:59.161869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5747d1-9e62-4ff2-97da-67d3cf501f23 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.162255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5747d1-9e62-4ff2-97da-67d3cf501f23 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.162630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5747d1-9e62-4ff2-97da-67d3cf501f23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.169382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6140c800-1e1f-456d-bc91-be43ce8cbf24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.170016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6140c800-1e1f-456d-bc91-be43ce8cbf24 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.170351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6140c800-1e1f-456d-bc91-be43ce8cbf24 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.170819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6140c800-1e1f-456d-bc91-be43ce8cbf24 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.171492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6140c800-1e1f-456d-bc91-be43ce8cbf24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.173143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6140c800-1e1f-456d-bc91-be43ce8cbf24 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.174729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:59.189171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 788/1614] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:59 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:59.190740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afd9102-e9ed-4764-ab9a-4979af04fb2e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.191103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afd9102-e9ed-4764-ab9a-4979af04fb2e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.191748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afd9102-e9ed-4764-ab9a-4979af04fb2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.193470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-af5747d1-9e62-4ff2-97da-67d3cf501f23 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:59.194201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-af5747d1-9e62-4ff2-97da-67d3cf501f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:59.204080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 827/1615] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:59.204720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5747d1-9e62-4ff2-97da-67d3cf501f23 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.205989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-af5747d1-9e62-4ff2-97da-67d3cf501f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:59.212122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-6afd9102-e9ed-4764-ab9a-4979af04fb2e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:59.213479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6afd9102-e9ed-4764-ab9a-4979af04fb2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:59.220798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5747d1-9e62-4ff2-97da-67d3cf501f23 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.222750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 789/1616] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:59.225365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a259f7-23de-4486-ab6f-595f481805da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.226259 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a259f7-23de-4486-ab6f-595f481805da admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.226543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a259f7-23de-4486-ab6f-595f481805da admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.226811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a259f7-23de-4486-ab6f-595f481805da admin admin] PATH_INFO: `/v3/projects/eb3ff1d4a6b541f7be2fa62fe0d686a5/users/fbf26518f49343ec925f4f7fd3d78c80/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.227089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afd9102-e9ed-4764-ab9a-4979af04fb2e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.229084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6afd9102-e9ed-4764-ab9a-4979af04fb2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:59.245354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a259f7-23de-4486-ab6f-595f481805da admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb3ff1d4a6b541f7be2fa62fe0d686a5, user_id=fbf26518f49343ec925f4f7fd3d78c80, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.248631 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a259f7-23de-4486-ab6f-595f481805da admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.265346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 828/1617] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:59 2022] PUT /identity/v3/projects/eb3ff1d4a6b541f7be2fa62fe0d686a5/users/fbf26518f49343ec925f4f7fd3d78c80/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:59.266181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a259f7-23de-4486-ab6f-595f481805da admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.303665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afd9102-e9ed-4764-ab9a-4979af04fb2e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.356584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a259f7-23de-4486-ab6f-595f481805da admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.361539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2daa5f7-98f1-407d-afc8-1bb39534de80 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.362214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2daa5f7-98f1-407d-afc8-1bb39534de80 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.362471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043b5f55-6a7e-4834-9f3c-66aa10e2fc29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.362648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2daa5f7-98f1-407d-afc8-1bb39534de80 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.362817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2daa5f7-98f1-407d-afc8-1bb39534de80 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.363229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043b5f55-6a7e-4834-9f3c-66aa10e2fc29 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.363539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043b5f55-6a7e-4834-9f3c-66aa10e2fc29 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.363848 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043b5f55-6a7e-4834-9f3c-66aa10e2fc29 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.364615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043b5f55-6a7e-4834-9f3c-66aa10e2fc29 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:51:59.364921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043b5f55-6a7e-4834-9f3c-66aa10e2fc29 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.366946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043b5f55-6a7e-4834-9f3c-66aa10e2fc29 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.372586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2daa5f7-98f1-407d-afc8-1bb39534de80 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.374598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2daa5f7-98f1-407d-afc8-1bb39534de80 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.384191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 829/1618] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:51:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:51:59.385451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfecf2e-45ef-4a2b-9280-7bb67804e57c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.385730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfecf2e-45ef-4a2b-9280-7bb67804e57c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.386003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfecf2e-45ef-4a2b-9280-7bb67804e57c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.390448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 790/1619] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:51:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:51:59.391209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2daa5f7-98f1-407d-afc8-1bb39534de80 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.402590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2daa5f7-98f1-407d-afc8-1bb39534de80 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.408581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c25ad9-a0d7-492c-91b5-d32f94ee002e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.409186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c25ad9-a0d7-492c-91b5-d32f94ee002e admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.409467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c25ad9-a0d7-492c-91b5-d32f94ee002e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.409803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c25ad9-a0d7-492c-91b5-d32f94ee002e admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.410466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c25ad9-a0d7-492c-91b5-d32f94ee002e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.412380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c25ad9-a0d7-492c-91b5-d32f94ee002e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.419939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ecfecf2e-45ef-4a2b-9280-7bb67804e57c None None] MFA Rules not processed for user `fbf26518f49343ec925f4f7fd3d78c80`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:59.421850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ecfecf2e-45ef-4a2b-9280-7bb67804e57c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:51:59.449675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 791/1620] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/projects => generated 363 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:59.457373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c25ad9-a0d7-492c-91b5-d32f94ee002e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.466583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:59.522824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 830/1621] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:51:59.528385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfecf2e-45ef-4a2b-9280-7bb67804e57c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.543621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfecf2e-45ef-4a2b-9280-7bb67804e57c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.552374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2782abbd-541e-488c-a1e4-36c581562a24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.553145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782abbd-541e-488c-a1e4-36c581562a24 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.553271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c25ad9-a0d7-492c-91b5-d32f94ee002e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.553425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782abbd-541e-488c-a1e4-36c581562a24 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.553696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782abbd-541e-488c-a1e4-36c581562a24 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.554333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2782abbd-541e-488c-a1e4-36c581562a24 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.556339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2782abbd-541e-488c-a1e4-36c581562a24 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.558445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fb8647-2774-4fda-9acd-a1c793ae699d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.559204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb8647-2774-4fda-9acd-a1c793ae699d admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.559492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb8647-2774-4fda-9acd-a1c793ae699d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.559857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb8647-2774-4fda-9acd-a1c793ae699d admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.560786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fb8647-2774-4fda-9acd-a1c793ae699d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.563362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fb8647-2774-4fda-9acd-a1c793ae699d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.591103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 831/1622] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/projects => generated 377 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:51:59.594322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 792/1623] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/users => generated 282 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:59.598493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fb8647-2774-4fda-9acd-a1c793ae699d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.598668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2782abbd-541e-488c-a1e4-36c581562a24 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.614024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fb8647-2774-4fda-9acd-a1c793ae699d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.633994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:51:59.686349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2782abbd-541e-488c-a1e4-36c581562a24 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.692621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f252bec-fabc-4ba3-aacd-57a3a65d9936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.693795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f252bec-fabc-4ba3-aacd-57a3a65d9936 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.694198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f252bec-fabc-4ba3-aacd-57a3a65d9936 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.694570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f252bec-fabc-4ba3-aacd-57a3a65d9936 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.695499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f252bec-fabc-4ba3-aacd-57a3a65d9936 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.697509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f252bec-fabc-4ba3-aacd-57a3a65d9936 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.718624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 832/1624] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:59 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:51:59.719835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f252bec-fabc-4ba3-aacd-57a3a65d9936 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.732854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f252bec-fabc-4ba3-aacd-57a3a65d9936 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.739290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6fbc9c-7735-417a-9083-4e75b4c8515a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.740062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fbc9c-7735-417a-9083-4e75b4c8515a admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.740733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fbc9c-7735-417a-9083-4e75b4c8515a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.741295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fbc9c-7735-417a-9083-4e75b4c8515a admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.742065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6fbc9c-7735-417a-9083-4e75b4c8515a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.747138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6fbc9c-7735-417a-9083-4e75b4c8515a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.766913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 833/1625] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:59 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:51:59.768034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6fbc9c-7735-417a-9083-4e75b4c8515a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.780230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6fbc9c-7735-417a-9083-4e75b4c8515a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.790085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802739e2-8e06-455e-a0e5-bbf41578fdee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.790622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802739e2-8e06-455e-a0e5-bbf41578fdee admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.791479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802739e2-8e06-455e-a0e5-bbf41578fdee admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.792017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802739e2-8e06-455e-a0e5-bbf41578fdee admin admin] PATH_INFO: `/v3/projects/80ab191d563a4156bd28694bbb12cb65/users/519b2d40b49f48ca9f312cc70625fa2a/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.812656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802739e2-8e06-455e-a0e5-bbf41578fdee admin admin] RBAC: Authorizing `identity:create_grant(project_id=80ab191d563a4156bd28694bbb12cb65, user_id=519b2d40b49f48ca9f312cc70625fa2a, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.816660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802739e2-8e06-455e-a0e5-bbf41578fdee admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.838401 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 834/1626] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:51:59 2022] PUT /identity/v3/projects/80ab191d563a4156bd28694bbb12cb65/users/519b2d40b49f48ca9f312cc70625fa2a/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:51:59.850587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3c5316-6b16-4c5e-a4c5-96efd6e7af27 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.851198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3c5316-6b16-4c5e-a4c5-96efd6e7af27 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.851676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3c5316-6b16-4c5e-a4c5-96efd6e7af27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.857434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44594360-fd75-41a1-a7cd-630e35c455d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:51:59.858295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44594360-fd75-41a1-a7cd-630e35c455d5 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:51:59.858857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44594360-fd75-41a1-a7cd-630e35c455d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:51:59.859238 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44594360-fd75-41a1-a7cd-630e35c455d5 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:51:59.859979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44594360-fd75-41a1-a7cd-630e35c455d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:51:59.863582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44594360-fd75-41a1-a7cd-630e35c455d5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:51:59.891622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 793/1627] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/users => generated 295 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:51:59.898957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44594360-fd75-41a1-a7cd-630e35c455d5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:51:59.913849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44594360-fd75-41a1-a7cd-630e35c455d5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:51:59.916558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-2a3c5316-6b16-4c5e-a4c5-96efd6e7af27 None None] MFA Rules not processed for user `519b2d40b49f48ca9f312cc70625fa2a`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:51:59.917980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2a3c5316-6b16-4c5e-a4c5-96efd6e7af27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:00.025648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a3c9a-e617-4f11-9915-5a92ff6d9816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.026277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a3c9a-e617-4f11-9915-5a92ff6d9816 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.026556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a3c9a-e617-4f11-9915-5a92ff6d9816 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.026826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a3c9a-e617-4f11-9915-5a92ff6d9816 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.027478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a3c9a-e617-4f11-9915-5a92ff6d9816 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.029106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a3c9a-e617-4f11-9915-5a92ff6d9816 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.043242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:00.047595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 794/1628] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:51:59 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:00.049463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a3c9a-e617-4f11-9915-5a92ff6d9816 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.053097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 835/1629] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:51:59 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:00.059554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3c5316-6b16-4c5e-a4c5-96efd6e7af27 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.066228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a3c9a-e617-4f11-9915-5a92ff6d9816 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.072822 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52582c6c-5429-401f-af15-b119065854c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.073624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52582c6c-5429-401f-af15-b119065854c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.073909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52582c6c-5429-401f-af15-b119065854c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.074182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52582c6c-5429-401f-af15-b119065854c3 admin admin] PATH_INFO: `/v3/projects/4ae3d4f3769e4045b615b69517b75944/users/26c36a11fd0442b5a0afd09dfe4396f4/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.075952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3c5316-6b16-4c5e-a4c5-96efd6e7af27 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.082617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4486c578-163f-499e-91b9-71d59242f2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.083816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4486c578-163f-499e-91b9-71d59242f2c1 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.084245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4486c578-163f-499e-91b9-71d59242f2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.084621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4486c578-163f-499e-91b9-71d59242f2c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.086136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4486c578-163f-499e-91b9-71d59242f2c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.087826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4486c578-163f-499e-91b9-71d59242f2c1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.090423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52582c6c-5429-401f-af15-b119065854c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ae3d4f3769e4045b615b69517b75944, user_id=26c36a11fd0442b5a0afd09dfe4396f4, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.094098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52582c6c-5429-401f-af15-b119065854c3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.115086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 795/1630] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:00 2022] PUT /identity/v3/projects/4ae3d4f3769e4045b615b69517b75944/users/26c36a11fd0442b5a0afd09dfe4396f4/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:00.120948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52582c6c-5429-401f-af15-b119065854c3 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.136073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 836/1631] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:00 2022] POST /identity/v3/projects => generated 363 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:00.139123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4486c578-163f-499e-91b9-71d59242f2c1 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.247540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52582c6c-5429-401f-af15-b119065854c3 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.254390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d5fd4-aff3-4906-b7cb-f4862150ee55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.255332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d5fd4-aff3-4906-b7cb-f4862150ee55 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.255610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d5fd4-aff3-4906-b7cb-f4862150ee55 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.255944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d5fd4-aff3-4906-b7cb-f4862150ee55 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.257125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d5fd4-aff3-4906-b7cb-f4862150ee55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.259090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4486c578-163f-499e-91b9-71d59242f2c1 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.259227 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d5fd4-aff3-4906-b7cb-f4862150ee55 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.265333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47317133-9dd7-469a-95c2-6abb0d777014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.266053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47317133-9dd7-469a-95c2-6abb0d777014 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.266430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47317133-9dd7-469a-95c2-6abb0d777014 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.266799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47317133-9dd7-469a-95c2-6abb0d777014 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.267670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47317133-9dd7-469a-95c2-6abb0d777014 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.269778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47317133-9dd7-469a-95c2-6abb0d777014 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.276117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 796/1632] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:00 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:00.303120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 837/1633] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:00 2022] POST /identity/v3/users => generated 288 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:00.303337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d5fd4-aff3-4906-b7cb-f4862150ee55 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.305161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47317133-9dd7-469a-95c2-6abb0d777014 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.317402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47317133-9dd7-469a-95c2-6abb0d777014 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.326030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b10b9b-c460-47bb-b26d-19da240284eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.327553 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b10b9b-c460-47bb-b26d-19da240284eb admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.331303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b10b9b-c460-47bb-b26d-19da240284eb admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.331714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b10b9b-c460-47bb-b26d-19da240284eb admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.333079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b10b9b-c460-47bb-b26d-19da240284eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.336105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d5fd4-aff3-4906-b7cb-f4862150ee55 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.338200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b10b9b-c460-47bb-b26d-19da240284eb admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.342938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a50242-899a-4606-ad35-176562a8fdda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.344114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a50242-899a-4606-ad35-176562a8fdda admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.344569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a50242-899a-4606-ad35-176562a8fdda admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.345078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a50242-899a-4606-ad35-176562a8fdda admin admin] PATH_INFO: `/v3/projects/4ae3d4f3769e4045b615b69517b75944/users/26c36a11fd0442b5a0afd09dfe4396f4/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.347638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a50242-899a-4606-ad35-176562a8fdda admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ae3d4f3769e4045b615b69517b75944, user_id=26c36a11fd0442b5a0afd09dfe4396f4, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.351744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a50242-899a-4606-ad35-176562a8fdda admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.361005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:00.369919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 838/1634] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:00 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:00.371603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b10b9b-c460-47bb-b26d-19da240284eb admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.386531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 797/1635] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:00 2022] PUT /identity/v3/projects/4ae3d4f3769e4045b615b69517b75944/users/26c36a11fd0442b5a0afd09dfe4396f4/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:00.386531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd004e2-4ef0-4382-a7e9-2312a7954b88 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.386531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd004e2-4ef0-4382-a7e9-2312a7954b88 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.386804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b10b9b-c460-47bb-b26d-19da240284eb admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.386889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd004e2-4ef0-4382-a7e9-2312a7954b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.444350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5cd004e2-4ef0-4382-a7e9-2312a7954b88 None None] MFA Rules not processed for user `26c36a11fd0442b5a0afd09dfe4396f4`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:00.447287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5cd004e2-4ef0-4382-a7e9-2312a7954b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:00.474424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.475165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.475509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.476400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] PATH_INFO: `/v3/projects/7cfeee4f4b554e5b93906cc86d5475e1/users/128dedcde6fb47bb92ceee83fe2f280c/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.495861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cfeee4f4b554e5b93906cc86d5475e1, user_id=128dedcde6fb47bb92ceee83fe2f280c, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.498668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.520261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 839/1636] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:00 2022] PUT /identity/v3/projects/7cfeee4f4b554e5b93906cc86d5475e1/users/128dedcde6fb47bb92ceee83fe2f280c/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:00.520261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.523094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:00.560002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 798/1637] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:00 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:00.561500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd004e2-4ef0-4382-a7e9-2312a7954b88 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.579872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:00.588610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd004e2-4ef0-4382-a7e9-2312a7954b88 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.608221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712cac65-a3a7-4262-972f-54d041d860bc admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.620515 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.621037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.621316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.621589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] PATH_INFO: `/v3/projects/46451dd7f7114302ac9cf1a9a46cb8c9` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.623243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=46451dd7f7114302ac9cf1a9a46cb8c9)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.626541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.676062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.676977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.677354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.677702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.678429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.680386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.685923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46451dd7f7114302ac9cf1a9a46cb8c9', 'request_id': 'req-e339afab-4368-436a-80c3-4aac4d6fce4f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:00.694876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 799/1638] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:00 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:00.694876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.694876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:00.696727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46451dd7f7114302ac9cf1a9a46cb8c9', 'request_id': 'req-e339afab-4368-436a-80c3-4aac4d6fce4f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:00.709587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46451dd7f7114302ac9cf1a9a46cb8c9', 'request_id': 'req-e339afab-4368-436a-80c3-4aac4d6fce4f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:00.725424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] The token cache is being invalidate because project 46451dd7f7114302ac9cf1a9a46cb8c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:00.725891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:00.726636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:00.728909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 840/1639] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:00 2022] DELETE /identity/v3/projects/46451dd7f7114302ac9cf1a9a46cb8c9 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:00.729711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.731900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:00.770194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be0d42e-f008-4b7e-b20e-aefca44203a1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.823911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339afab-4368-436a-80c3-4aac4d6fce4f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.825711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:00.829827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.830724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.830943 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.831218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] PATH_INFO: `/v3/projects/7cfeee4f4b554e5b93906cc86d5475e1/users/128dedcde6fb47bb92ceee83fe2f280c/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.833290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cfeee4f4b554e5b93906cc86d5475e1, user_id=128dedcde6fb47bb92ceee83fe2f280c, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.836069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.844498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05bd237-4a54-45a4-bc4c-dad296a98c59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.845103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05bd237-4a54-45a4-bc4c-dad296a98c59 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.845839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05bd237-4a54-45a4-bc4c-dad296a98c59 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.845839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05bd237-4a54-45a4-bc4c-dad296a98c59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.847641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e05bd237-4a54-45a4-bc4c-dad296a98c59 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:00.847776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 841/1640] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:00 2022] PUT /identity/v3/projects/7cfeee4f4b554e5b93906cc86d5475e1/users/128dedcde6fb47bb92ceee83fe2f280c/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:00.848424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:00.849948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:00.918005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05bd237-4a54-45a4-bc4c-dad296a98c59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.919360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05bd237-4a54-45a4-bc4c-dad296a98c59 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.924352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1a2c55-ab1f-464f-bc77-9b9f658951d2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:00.929171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:00.929731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.930174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.930297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 800/1641] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:00 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:00.930404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] PATH_INFO: `/v3/users/9f7aaa388a4b4014aca92d036375013f` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.931291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f06ea9-19d9-4912-ac8f-76f1022fa0f2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:00.931576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f06ea9-19d9-4912-ac8f-76f1022fa0f2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:00.931847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f06ea9-19d9-4912-ac8f-76f1022fa0f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:00.931950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f7aaa388a4b4014aca92d036375013f)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:00.933818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:00.938885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:00.963475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b0f06ea9-19d9-4912-ac8f-76f1022fa0f2 None None] MFA Rules not processed for user `128dedcde6fb47bb92ceee83fe2f280c`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:00.964746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b0f06ea9-19d9-4912-ac8f-76f1022fa0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:01.007076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f7aaa388a4b4014aca92d036375013f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.016845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f7aaa388a4b4014aca92d036375013f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.028998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f7aaa388a4b4014aca92d036375013f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.029730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f7aaa388a4b4014aca92d036375013f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.031516 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f7aaa388a4b4014aca92d036375013f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.045836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 842/1642] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:00 2022] DELETE /identity/v3/users/9f7aaa388a4b4014aca92d036375013f => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:01.046746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:01.047899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:01.060720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 801/1643] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:00 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:01.072590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f06ea9-19d9-4912-ac8f-76f1022fa0f2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:01.074279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b0f06ea9-19d9-4912-ac8f-76f1022fa0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:01.118128 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0c60b3-9c94-46d0-8f93-0fe2af118c1e admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:01.122502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:01.123097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:01.123584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:01.123969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:01.126290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:01.141632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f06ea9-19d9-4912-ac8f-76f1022fa0f2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:01.146652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0071a03-14a7-49a5-8783-cc9a30f5e5ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:01.147353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0071a03-14a7-49a5-8783-cc9a30f5e5ff service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:01.147562 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0071a03-14a7-49a5-8783-cc9a30f5e5ff service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:01.147804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0071a03-14a7-49a5-8783-cc9a30f5e5ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:01.183815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:01.243196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:01.244773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:01.250877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0071a03-14a7-49a5-8783-cc9a30f5e5ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:01.253485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0071a03-14a7-49a5-8783-cc9a30f5e5ff service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:01.269146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 843/1644] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:01 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:01.274302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 802/1645] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:01 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:01.369589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:01.371295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:01.438541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0071a03-14a7-49a5-8783-cc9a30f5e5ff service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:01.448312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0071a03-14a7-49a5-8783-cc9a30f5e5ff service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:01.453633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:01.454244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:01.454639 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:01.454907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:01.457402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:01.464632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:01.499661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a15f3aa-07d0-46fe-97bb-2d8f7871ade9 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:01.506196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:01.507108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:01.507457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:01.508240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] PATH_INFO: `/v3/projects/ae13cea6133c40878b57aaf15ca0e8f4` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:01.509715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae13cea6133c40878b57aaf15ca0e8f4)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:01.511753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:01.562638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:01.564240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:01.583561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 803/1646] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:01 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:01.597455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae13cea6133c40878b57aaf15ca0e8f4', 'request_id': 'req-63e450d5-5e65-449b-bfae-c0e8809a16f9'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.609724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae13cea6133c40878b57aaf15ca0e8f4', 'request_id': 'req-63e450d5-5e65-449b-bfae-c0e8809a16f9'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.619749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae13cea6133c40878b57aaf15ca0e8f4', 'request_id': 'req-63e450d5-5e65-449b-bfae-c0e8809a16f9'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.637040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] The token cache is being invalidate because project ae13cea6133c40878b57aaf15ca0e8f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:01.637503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.638785 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.641818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 844/1647] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:01 2022] DELETE /identity/v3/projects/ae13cea6133c40878b57aaf15ca0e8f4 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:01.645830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:01.663596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0090f2c-a5ee-4421-9f92-f2fce409e863 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:01.814597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:01.815426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:01.815535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:01.815784 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] PATH_INFO: `/v3/users/b7affd8d7752418895bf0b7ea5036232` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:01.817188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7affd8d7752418895bf0b7ea5036232)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:01.819784 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:01.873950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7affd8d7752418895bf0b7ea5036232'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.881697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7affd8d7752418895bf0b7ea5036232'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.887171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:01.899248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7affd8d7752418895bf0b7ea5036232'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.900065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7affd8d7752418895bf0b7ea5036232'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.900775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17ae0133-b1ee-4e72-a382-bfe8f41e31d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7affd8d7752418895bf0b7ea5036232'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:01.915387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 804/1648] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:01 2022] DELETE /identity/v3/users/b7affd8d7752418895bf0b7ea5036232 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:02.213770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:02.215364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:02.233402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e450d5-5e65-449b-bfae-c0e8809a16f9 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:02.293004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:02.293643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.293923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.294196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] PATH_INFO: `/v3/projects/70e3997385ec481a9052a61d1157bbc6` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.295315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=70e3997385ec481a9052a61d1157bbc6)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:02.297312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:02.346726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '70e3997385ec481a9052a61d1157bbc6', 'request_id': 'req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:02.353045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '70e3997385ec481a9052a61d1157bbc6', 'request_id': 'req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:02.361598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '70e3997385ec481a9052a61d1157bbc6', 'request_id': 'req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:02.368207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:02.374419 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] The token cache is being invalidate because project 70e3997385ec481a9052a61d1157bbc6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:02.374765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:02.375461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-efd75a56-dd50-48f0-b4ac-a821a87cbf1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:02.379412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 845/1649] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:02 2022] DELETE /identity/v3/projects/70e3997385ec481a9052a61d1157bbc6 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:02.386167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b6e1f2-dba7-45e2-9e8b-4951543d1b37 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.386737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b6e1f2-dba7-45e2-9e8b-4951543d1b37 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.387017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b6e1f2-dba7-45e2-9e8b-4951543d1b37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.408284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-71b6e1f2-dba7-45e2-9e8b-4951543d1b37 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:02.409536 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-71b6e1f2-dba7-45e2-9e8b-4951543d1b37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:02.473748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 805/1650] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:02.481778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781f7f57-17be-406d-904e-606e156677a9 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.482060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781f7f57-17be-406d-904e-606e156677a9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.482331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781f7f57-17be-406d-904e-606e156677a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.503920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-781f7f57-17be-406d-904e-606e156677a9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:02.505171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-781f7f57-17be-406d-904e-606e156677a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:02.513445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 846/1651] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:02.518022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b6e1f2-dba7-45e2-9e8b-4951543d1b37 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:02.527813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b6e1f2-dba7-45e2-9e8b-4951543d1b37 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:02.533332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:02.534014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.534294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.534576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.535259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:02.535564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:02.537596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:02.550913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 806/1652] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:02.558436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781f7f57-17be-406d-904e-606e156677a9 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:02.568881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781f7f57-17be-406d-904e-606e156677a9 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:02.573701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1375258-7545-480a-806d-010b711e6d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:02.574305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1375258-7545-480a-806d-010b711e6d3b admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.574584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1375258-7545-480a-806d-010b711e6d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.574854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1375258-7545-480a-806d-010b711e6d3b admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.575513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1375258-7545-480a-806d-010b711e6d3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:02.577581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1375258-7545-480a-806d-010b711e6d3b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:02.608670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 847/1653] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:02 2022] POST /identity/v3/projects => generated 401 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:02.613202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:02.624466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e75c33-d0bd-4566-8f0b-791f9e66e8bc admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:02.685992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32b6af4-75e9-4965-97bb-d52c0864a138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:02.686675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32b6af4-75e9-4965-97bb-d52c0864a138 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.686963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32b6af4-75e9-4965-97bb-d52c0864a138 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.687232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32b6af4-75e9-4965-97bb-d52c0864a138 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.687907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32b6af4-75e9-4965-97bb-d52c0864a138 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:02.689827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32b6af4-75e9-4965-97bb-d52c0864a138 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:02.713691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 807/1654] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:02 2022] POST /identity/v3/users => generated 301 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:02.718497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1375258-7545-480a-806d-010b711e6d3b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:02.730081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1375258-7545-480a-806d-010b711e6d3b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:02.734962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f99101f-4d55-4405-8185-59e7fd7dee64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:02.735570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f99101f-4d55-4405-8185-59e7fd7dee64 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.735880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f99101f-4d55-4405-8185-59e7fd7dee64 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.736183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f99101f-4d55-4405-8185-59e7fd7dee64 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.736835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f99101f-4d55-4405-8185-59e7fd7dee64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:02.738423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f99101f-4d55-4405-8185-59e7fd7dee64 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:02.753349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 848/1655] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:02 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:02.754666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32b6af4-75e9-4965-97bb-d52c0864a138 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:02.766246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32b6af4-75e9-4965-97bb-d52c0864a138 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:02.771134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0beca7d-b0b9-42f6-ab8d-ccdd5345ff1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:02.771811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0beca7d-b0b9-42f6-ab8d-ccdd5345ff1c admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.772175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0beca7d-b0b9-42f6-ab8d-ccdd5345ff1c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.772518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0beca7d-b0b9-42f6-ab8d-ccdd5345ff1c admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.773225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0beca7d-b0b9-42f6-ab8d-ccdd5345ff1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:02.775001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0beca7d-b0b9-42f6-ab8d-ccdd5345ff1c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:02.790201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 808/1656] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:02 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:02.797349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f99101f-4d55-4405-8185-59e7fd7dee64 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:02.809417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f99101f-4d55-4405-8185-59e7fd7dee64 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:02.814906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28103b5-e1d1-45c9-8497-adde7f142a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:02.815060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28103b5-e1d1-45c9-8497-adde7f142a53 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.815263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28103b5-e1d1-45c9-8497-adde7f142a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.815541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28103b5-e1d1-45c9-8497-adde7f142a53 admin admin] PATH_INFO: `/v3/projects/bfd77a0fc47a4e6fac66f7f9ccad631f/users/6991a40984164029a4574c57fe3008f1/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.837022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28103b5-e1d1-45c9-8497-adde7f142a53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfd77a0fc47a4e6fac66f7f9ccad631f, user_id=6991a40984164029a4574c57fe3008f1, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:02.839921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28103b5-e1d1-45c9-8497-adde7f142a53 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:02.880585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9284359e-07fe-427b-9a76-8592f720521e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:02.880904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9284359e-07fe-427b-9a76-8592f720521e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:02.881217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9284359e-07fe-427b-9a76-8592f720521e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:02.881379 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 849/1657] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:02 2022] PUT /identity/v3/projects/bfd77a0fc47a4e6fac66f7f9ccad631f/users/6991a40984164029a4574c57fe3008f1/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:02.920434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9284359e-07fe-427b-9a76-8592f720521e None None] MFA Rules not processed for user `6991a40984164029a4574c57fe3008f1`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:02.925616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9284359e-07fe-427b-9a76-8592f720521e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:03.007425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:03.050921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 809/1658] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:02 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Oct 24 08:52:03.055396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28103b5-e1d1-45c9-8497-adde7f142a53 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:03.152233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28103b5-e1d1-45c9-8497-adde7f142a53 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:03.158466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934506f-2257-4098-8455-ba15a3265de1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:03.159080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934506f-2257-4098-8455-ba15a3265de1 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.159365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934506f-2257-4098-8455-ba15a3265de1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.159638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934506f-2257-4098-8455-ba15a3265de1 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:03.160305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3934506f-2257-4098-8455-ba15a3265de1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:03.162216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3934506f-2257-4098-8455-ba15a3265de1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:03.192692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 850/1659] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:03 2022] POST /identity/v3/projects => generated 399 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:03.196528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9284359e-07fe-427b-9a76-8592f720521e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:03.210452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9284359e-07fe-427b-9a76-8592f720521e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:03.284174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf669b29-4577-4ab6-ae63-26f061458458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:03.285327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf669b29-4577-4ab6-ae63-26f061458458 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.285845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf669b29-4577-4ab6-ae63-26f061458458 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.286387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf669b29-4577-4ab6-ae63-26f061458458 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:03.287227 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf669b29-4577-4ab6-ae63-26f061458458 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:03.289528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf669b29-4577-4ab6-ae63-26f061458458 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:03.329395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 810/1660] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:03 2022] POST /identity/v3/users => generated 306 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:03.333344 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934506f-2257-4098-8455-ba15a3265de1 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:03.359500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934506f-2257-4098-8455-ba15a3265de1 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:03.359500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314b64b7-b9b4-46c0-bb77-eb5d98392ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:03.360509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314b64b7-b9b4-46c0-bb77-eb5d98392ce3 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.360509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314b64b7-b9b4-46c0-bb77-eb5d98392ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.360722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314b64b7-b9b4-46c0-bb77-eb5d98392ce3 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:03.361211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314b64b7-b9b4-46c0-bb77-eb5d98392ce3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:03.363064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314b64b7-b9b4-46c0-bb77-eb5d98392ce3 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:03.380991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 851/1661] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:03 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:03.382527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf669b29-4577-4ab6-ae63-26f061458458 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:03.398475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf669b29-4577-4ab6-ae63-26f061458458 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:03.406073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c8d56-c1fc-4ba7-8b95-253a5fcb70d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:03.406472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c8d56-c1fc-4ba7-8b95-253a5fcb70d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.406783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c8d56-c1fc-4ba7-8b95-253a5fcb70d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.407026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c8d56-c1fc-4ba7-8b95-253a5fcb70d4 admin admin] PATH_INFO: `/v3/projects/d0018036835e4b0bac9de1eccd34dd22/users/53285422cfa54c30814540887f3d710a/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:03.424023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64c8d56-c1fc-4ba7-8b95-253a5fcb70d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0018036835e4b0bac9de1eccd34dd22, user_id=53285422cfa54c30814540887f3d710a, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:03.426773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64c8d56-c1fc-4ba7-8b95-253a5fcb70d4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:03.441409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 811/1662] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:03 2022] PUT /identity/v3/projects/d0018036835e4b0bac9de1eccd34dd22/users/53285422cfa54c30814540887f3d710a/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:03.445512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314b64b7-b9b4-46c0-bb77-eb5d98392ce3 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:03.463275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314b64b7-b9b4-46c0-bb77-eb5d98392ce3 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:03.543939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7307eb1d-1290-48f5-a69c-6ecc3391ad1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:03.544706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7307eb1d-1290-48f5-a69c-6ecc3391ad1d admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.545194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7307eb1d-1290-48f5-a69c-6ecc3391ad1d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.546115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7307eb1d-1290-48f5-a69c-6ecc3391ad1d admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:03.546952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7307eb1d-1290-48f5-a69c-6ecc3391ad1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:03.548715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7307eb1d-1290-48f5-a69c-6ecc3391ad1d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:03.561905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 852/1663] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:03 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:03.565912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c8d56-c1fc-4ba7-8b95-253a5fcb70d4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:03.587781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c8d56-c1fc-4ba7-8b95-253a5fcb70d4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:03.600082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:03.600967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.601780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.602120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] PATH_INFO: `/v3/projects/d0018036835e4b0bac9de1eccd34dd22/users/53285422cfa54c30814540887f3d710a/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:03.604539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0018036835e4b0bac9de1eccd34dd22, user_id=53285422cfa54c30814540887f3d710a, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:03.607435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:03.619739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 812/1664] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:03 2022] PUT /identity/v3/projects/d0018036835e4b0bac9de1eccd34dd22/users/53285422cfa54c30814540887f3d710a/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:03.631375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7396cdbf-bffa-4624-a3b8-8413b8f8d192 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.631662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7396cdbf-bffa-4624-a3b8-8413b8f8d192 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.631941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7396cdbf-bffa-4624-a3b8-8413b8f8d192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:03.640830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:03.666222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-7396cdbf-bffa-4624-a3b8-8413b8f8d192 None None] MFA Rules not processed for user `53285422cfa54c30814540887f3d710a`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:03.667421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7396cdbf-bffa-4624-a3b8-8413b8f8d192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:03.739657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 853/1665] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:03 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:03.752432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:03.754109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:03.814123 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2442bd-0c13-47bf-ad26-307a166201ac admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:03.819004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:03.819801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.820208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.820518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:03.832034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:03.833673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:03.844859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 813/1666] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:03 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:03.943635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7396cdbf-bffa-4624-a3b8-8413b8f8d192 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:03.960213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7396cdbf-bffa-4624-a3b8-8413b8f8d192 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:03.966486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ebd078-2a51-4a00-ae54-35c0a7c5aa1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:03.968427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ebd078-2a51-4a00-ae54-35c0a7c5aa1a service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:03.968978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ebd078-2a51-4a00-ae54-35c0a7c5aa1a service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:03.969841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ebd078-2a51-4a00-ae54-35c0a7c5aa1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:04.069191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ebd078-2a51-4a00-ae54-35c0a7c5aa1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:04.070633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ebd078-2a51-4a00-ae54-35c0a7c5aa1a service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:04.083435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 854/1667] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:03 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:04.516732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:04.519070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:04.562530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:04.597094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031d5c64-92cd-430a-af2a-ed1cc3e47831 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:04.601688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:04.602326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:04.602582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:04.602863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] PATH_INFO: `/v3/users/519b2d40b49f48ca9f312cc70625fa2a` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:04.603979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] RBAC: Authorizing `identity:delete_user(user_id=519b2d40b49f48ca9f312cc70625fa2a)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:04.605903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:04.657557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '519b2d40b49f48ca9f312cc70625fa2a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:04.666165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '519b2d40b49f48ca9f312cc70625fa2a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:04.679710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '519b2d40b49f48ca9f312cc70625fa2a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:04.680447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '519b2d40b49f48ca9f312cc70625fa2a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:04.681064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '519b2d40b49f48ca9f312cc70625fa2a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:04.696542 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 814/1668] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:04 2022] DELETE /identity/v3/users/519b2d40b49f48ca9f312cc70625fa2a => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:04.706372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e661ae0-99fb-4004-82a6-6168709d643a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:04.706864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e661ae0-99fb-4004-82a6-6168709d643a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:04.707242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e661ae0-99fb-4004-82a6-6168709d643a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:04.728965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-1e661ae0-99fb-4004-82a6-6168709d643a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:04.730171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1e661ae0-99fb-4004-82a6-6168709d643a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:04.789734 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 855/1669] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:04.800073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:04.801551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:04.861054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b96f0ac-f6b8-4ab3-bd2e-de859b94119c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:04.864685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfd413d-fd8d-4f00-a732-4df2c3d94015 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:04.865283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd413d-fd8d-4f00-a732-4df2c3d94015 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:04.865840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd413d-fd8d-4f00-a732-4df2c3d94015 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:04.866087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd413d-fd8d-4f00-a732-4df2c3d94015 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:04.876814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfd413d-fd8d-4f00-a732-4df2c3d94015 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:04.878241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfd413d-fd8d-4f00-a732-4df2c3d94015 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:04.888448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 815/1670] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:05.353158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e661ae0-99fb-4004-82a6-6168709d643a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:05.354887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1e661ae0-99fb-4004-82a6-6168709d643a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:05.384186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e661ae0-99fb-4004-82a6-6168709d643a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:05.389175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:05.390667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:05.390950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:05.391365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] PATH_INFO: `/v3/projects/80ab191d563a4156bd28694bbb12cb65` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:05.392584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80ab191d563a4156bd28694bbb12cb65)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:05.394478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:05.448039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:05.498858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80ab191d563a4156bd28694bbb12cb65', 'request_id': 'req-f4dfe224-f63f-41c4-b44f-dbd9beddf073'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.507170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80ab191d563a4156bd28694bbb12cb65', 'request_id': 'req-f4dfe224-f63f-41c4-b44f-dbd9beddf073'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.521876 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80ab191d563a4156bd28694bbb12cb65', 'request_id': 'req-f4dfe224-f63f-41c4-b44f-dbd9beddf073'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.544582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] The token cache is being invalidate because project 80ab191d563a4156bd28694bbb12cb65 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:05.544834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.546481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.549165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 856/1671] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:05 2022] DELETE /identity/v3/projects/80ab191d563a4156bd28694bbb12cb65 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:05.553658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfd413d-fd8d-4f00-a732-4df2c3d94015 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:05.561485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:05.570207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfd413d-fd8d-4f00-a732-4df2c3d94015 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:05.570787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:05.633051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:05.644478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:05.645282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:05.645889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:05.646174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] PATH_INFO: `/v3/users/128dedcde6fb47bb92ceee83fe2f280c` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:05.647331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=128dedcde6fb47bb92ceee83fe2f280c)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:05.649437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:05.658761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dfe224-f63f-41c4-b44f-dbd9beddf073 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:05.663661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:05.664322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:05.664605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:05.664881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] PATH_INFO: `/v3/users/40fb3a560b5e49f1940f77a357eaaf3b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:05.666027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] RBAC: Authorizing `identity:delete_user(user_id=40fb3a560b5e49f1940f77a357eaaf3b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:05.668217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:05.709143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '128dedcde6fb47bb92ceee83fe2f280c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.720868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '128dedcde6fb47bb92ceee83fe2f280c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.735479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40fb3a560b5e49f1940f77a357eaaf3b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.736744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '128dedcde6fb47bb92ceee83fe2f280c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.737822 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '128dedcde6fb47bb92ceee83fe2f280c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.738983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-10629fb5-8120-4434-9458-9f0b71eca2d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '128dedcde6fb47bb92ceee83fe2f280c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.743890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40fb3a560b5e49f1940f77a357eaaf3b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.754552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 816/1672] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:05 2022] DELETE /identity/v3/users/128dedcde6fb47bb92ceee83fe2f280c => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:05.760392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40fb3a560b5e49f1940f77a357eaaf3b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.761730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40fb3a560b5e49f1940f77a357eaaf3b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.768385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40fb3a560b5e49f1940f77a357eaaf3b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:05.780679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 857/1673] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:05 2022] DELETE /identity/v3/users/40fb3a560b5e49f1940f77a357eaaf3b => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:05.785084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbde664-632c-4ed5-89a0-000fe19e553a None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:05.785460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbde664-632c-4ed5-89a0-000fe19e553a None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:05.785643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbde664-632c-4ed5-89a0-000fe19e553a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:05.806371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5fbde664-632c-4ed5-89a0-000fe19e553a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:05.807492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5fbde664-632c-4ed5-89a0-000fe19e553a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:05.871583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 817/1674] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:05.881936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:05.883450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:05.929422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:05.954991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f283062d-ace9-460a-b553-5a9c90c8c20f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:05.959224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd9da0b-adc6-4e23-aa66-628d1d11c46b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:05.959980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9da0b-adc6-4e23-aa66-628d1d11c46b service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:05.960332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9da0b-adc6-4e23-aa66-628d1d11c46b service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:05.960604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9da0b-adc6-4e23-aa66-628d1d11c46b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:05.972835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd9da0b-adc6-4e23-aa66-628d1d11c46b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:05.978512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd9da0b-adc6-4e23-aa66-628d1d11c46b service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:05.984985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 858/1675] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:06.093012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbde664-632c-4ed5-89a0-000fe19e553a None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.094442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5fbde664-632c-4ed5-89a0-000fe19e553a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:06.107603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbde664-632c-4ed5-89a0-000fe19e553a None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.112553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.113193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.113475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.113992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] PATH_INFO: `/v3/projects/7cfeee4f4b554e5b93906cc86d5475e1` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.114890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cfeee4f4b554e5b93906cc86d5475e1)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.117730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.171604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cfeee4f4b554e5b93906cc86d5475e1', 'request_id': 'req-eee5395b-27e4-405e-aa15-e7c5849c75c5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.177667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cfeee4f4b554e5b93906cc86d5475e1', 'request_id': 'req-eee5395b-27e4-405e-aa15-e7c5849c75c5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.184549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cfeee4f4b554e5b93906cc86d5475e1', 'request_id': 'req-eee5395b-27e4-405e-aa15-e7c5849c75c5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.193702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:06.205933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] The token cache is being invalidate because project 7cfeee4f4b554e5b93906cc86d5475e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:06.206298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.207154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eee5395b-27e4-405e-aa15-e7c5849c75c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.208989 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 818/1676] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:06 2022] DELETE /identity/v3/projects/7cfeee4f4b554e5b93906cc86d5475e1 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:06.214862 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78be5068-386f-47a8-945d-8cdbb78fa3dd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.215149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78be5068-386f-47a8-945d-8cdbb78fa3dd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.215419 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78be5068-386f-47a8-945d-8cdbb78fa3dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.236826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-78be5068-386f-47a8-945d-8cdbb78fa3dd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:06.238045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-78be5068-386f-47a8-945d-8cdbb78fa3dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:06.300172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 859/1677] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:06.308580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d5d4d-7623-4fb0-ac2c-d32a34507335 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.308865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d5d4d-7623-4fb0-ac2c-d32a34507335 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.309134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d5d4d-7623-4fb0-ac2c-d32a34507335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.333283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78be5068-386f-47a8-945d-8cdbb78fa3dd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.336856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-78be5068-386f-47a8-945d-8cdbb78fa3dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:06.336965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e17d5d4d-7623-4fb0-ac2c-d32a34507335 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:06.338006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e17d5d4d-7623-4fb0-ac2c-d32a34507335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:06.347889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 819/1678] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:06.351874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17d5d4d-7623-4fb0-ac2c-d32a34507335 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.363116 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17d5d4d-7623-4fb0-ac2c-d32a34507335 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.363702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78be5068-386f-47a8-945d-8cdbb78fa3dd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.369898 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.370619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.370968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.371613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] PATH_INFO: `/v3/projects/150c629df022430da59901564c3e63bf` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.372808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=150c629df022430da59901564c3e63bf)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.372921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.373859 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.375296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.375556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.375662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.377884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:06.378322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.380607 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.400173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 820/1679] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:06.401945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.415354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066af024-ee69-4bae-a63a-b156860c6659 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.421844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbeccc4f-4a92-469c-be66-ecc2200af45b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.422758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeccc4f-4a92-469c-be66-ecc2200af45b admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.423018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeccc4f-4a92-469c-be66-ecc2200af45b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.423292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeccc4f-4a92-469c-be66-ecc2200af45b admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.424028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbeccc4f-4a92-469c-be66-ecc2200af45b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.426170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbeccc4f-4a92-469c-be66-ecc2200af45b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.445804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '150c629df022430da59901564c3e63bf', 'request_id': 'req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.454552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '150c629df022430da59901564c3e63bf', 'request_id': 'req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.459562 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 821/1680] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:06 2022] POST /identity/v3/projects => generated 381 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:06.464742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbeccc4f-4a92-469c-be66-ecc2200af45b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.467090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '150c629df022430da59901564c3e63bf', 'request_id': 'req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.481306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] The token cache is being invalidate because project 150c629df022430da59901564c3e63bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:06.481602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.482279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.484702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 860/1681] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:06 2022] DELETE /identity/v3/projects/150c629df022430da59901564c3e63bf => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:06.488248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.489696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:06.550105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbeccc4f-4a92-469c-be66-ecc2200af45b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.557675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.558567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.558930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.559202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.559883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.562411 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.579630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68a0ae6-92b1-4eda-84f7-f43f712bc8fe admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.581355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:06.590476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 822/1682] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:06 2022] POST /identity/v3/users => generated 291 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:06.594290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.595244 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.595534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.596243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] PATH_INFO: `/v3/users/fbf26518f49343ec925f4f7fd3d78c80` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.598365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbf26518f49343ec925f4f7fd3d78c80)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.600675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.604921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.606953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:06.621228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5003576-6d51-4a13-8859-4d03f44ddc3d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.628249 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a630c2-e760-49e5-85c7-8af6a79d2dd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.629185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a630c2-e760-49e5-85c7-8af6a79d2dd4 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.629775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a630c2-e760-49e5-85c7-8af6a79d2dd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.630883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a630c2-e760-49e5-85c7-8af6a79d2dd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.631874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a630c2-e760-49e5-85c7-8af6a79d2dd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.634673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a630c2-e760-49e5-85c7-8af6a79d2dd4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.650745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 823/1683] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:06 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:06.656021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a630c2-e760-49e5-85c7-8af6a79d2dd4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.677438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a630c2-e760-49e5-85c7-8af6a79d2dd4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.691997 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b561b795-ac04-49f4-a6f6-6f107ee1b68f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.694093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561b795-ac04-49f4-a6f6-6f107ee1b68f admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.694393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561b795-ac04-49f4-a6f6-6f107ee1b68f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.694977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561b795-ac04-49f4-a6f6-6f107ee1b68f admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.697256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b561b795-ac04-49f4-a6f6-6f107ee1b68f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.702996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b561b795-ac04-49f4-a6f6-6f107ee1b68f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.737053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 824/1684] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:06 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:06.741837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b561b795-ac04-49f4-a6f6-6f107ee1b68f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:06.757803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbf26518f49343ec925f4f7fd3d78c80'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.770441 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b561b795-ac04-49f4-a6f6-6f107ee1b68f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:06.775056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbf26518f49343ec925f4f7fd3d78c80'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.785275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:06.786002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.786287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.786568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] PATH_INFO: `/v3/projects/18b37d8b6ce64b7cbae756808a0caa5d/users/ec13ff540dd74041883081017fa6f49a/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:06.806630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbf26518f49343ec925f4f7fd3d78c80'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.807424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbf26518f49343ec925f4f7fd3d78c80'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.808183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0c9dafbf-003a-4637-9131-e936dd8a092f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbf26518f49343ec925f4f7fd3d78c80'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:06.828794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18b37d8b6ce64b7cbae756808a0caa5d, user_id=ec13ff540dd74041883081017fa6f49a, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:06.834176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 861/1685] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:06 2022] DELETE /identity/v3/users/fbf26518f49343ec925f4f7fd3d78c80 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:06.848938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:06.899525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 825/1686] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:06 2022] PUT /identity/v3/projects/18b37d8b6ce64b7cbae756808a0caa5d/users/ec13ff540dd74041883081017fa6f49a/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:06.912428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefbfa0f-742d-49d8-b27e-fbd56c82146d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:06.912710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefbfa0f-742d-49d8-b27e-fbd56c82146d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:06.913092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefbfa0f-742d-49d8-b27e-fbd56c82146d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.016995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fefbfa0f-742d-49d8-b27e-fbd56c82146d None None] MFA Rules not processed for user `ec13ff540dd74041883081017fa6f49a`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:07.016995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fefbfa0f-742d-49d8-b27e-fbd56c82146d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:07.180631 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 862/1687] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:06 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:07.184124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.189903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:07.298108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:07.303879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea83aa06-f443-4513-8697-44bca1c32a35 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:07.313049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbc833a-feba-49ff-96ec-c9fa5a6e5b0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:07.314050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc833a-feba-49ff-96ec-c9fa5a6e5b0b admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.314353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc833a-feba-49ff-96ec-c9fa5a6e5b0b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.315257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc833a-feba-49ff-96ec-c9fa5a6e5b0b admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.315396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbc833a-feba-49ff-96ec-c9fa5a6e5b0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:07.317564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbc833a-feba-49ff-96ec-c9fa5a6e5b0b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:07.354754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 826/1688] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:07 2022] POST /identity/v3/projects => generated 381 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:07.356386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefbfa0f-742d-49d8-b27e-fbd56c82146d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.376086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefbfa0f-742d-49d8-b27e-fbd56c82146d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:07.444371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:07.444959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.445240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.445513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.446156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:07.448180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:07.461472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:07.477730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 863/1689] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:07 2022] POST /identity/v3/users => generated 297 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:07.481506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbc833a-feba-49ff-96ec-c9fa5a6e5b0b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.487342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.489452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:07.496060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbc833a-feba-49ff-96ec-c9fa5a6e5b0b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:07.501667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824cfac6-2289-4855-bc29-0a2239ad07ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:07.502289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cfac6-2289-4855-bc29-0a2239ad07ab admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.502676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cfac6-2289-4855-bc29-0a2239ad07ab admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.503047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cfac6-2289-4855-bc29-0a2239ad07ab admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.503598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824cfac6-2289-4855-bc29-0a2239ad07ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:07.506527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824cfac6-2289-4855-bc29-0a2239ad07ab admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:07.507879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf3036-0db6-4722-814e-8640688cf89f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:07.513523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:07.515128 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.515519 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.515932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] PATH_INFO: `/v3/projects/eb3ff1d4a6b541f7be2fa62fe0d686a5` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.517350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb3ff1d4a6b541f7be2fa62fe0d686a5)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:07.519970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:07.543337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 827/1690] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:07 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:07.544433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824cfac6-2289-4855-bc29-0a2239ad07ab admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.557213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824cfac6-2289-4855-bc29-0a2239ad07ab admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:07.563286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5c72e-965a-445c-aee1-548d19338517 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:07.563979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5c72e-965a-445c-aee1-548d19338517 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.564385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5c72e-965a-445c-aee1-548d19338517 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.564664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5c72e-965a-445c-aee1-548d19338517 admin admin] PATH_INFO: `/v3/projects/df97399ef8cf4da089964120403e4f8f/users/63659bff07cc466a8aa6019951f39965/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.580792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb3ff1d4a6b541f7be2fa62fe0d686a5', 'request_id': 'req-1e734007-cc65-4116-9ca8-5f8510da8633'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.581993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5c72e-965a-445c-aee1-548d19338517 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df97399ef8cf4da089964120403e4f8f, user_id=63659bff07cc466a8aa6019951f39965, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:07.585927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5c72e-965a-445c-aee1-548d19338517 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:07.588353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb3ff1d4a6b541f7be2fa62fe0d686a5', 'request_id': 'req-1e734007-cc65-4116-9ca8-5f8510da8633'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.597074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb3ff1d4a6b541f7be2fa62fe0d686a5', 'request_id': 'req-1e734007-cc65-4116-9ca8-5f8510da8633'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.600843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 828/1691] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:07 2022] PUT /identity/v3/projects/df97399ef8cf4da089964120403e4f8f/users/63659bff07cc466a8aa6019951f39965/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:07.604850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5c72e-965a-445c-aee1-548d19338517 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.611491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] The token cache is being invalidate because project eb3ff1d4a6b541f7be2fa62fe0d686a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:07.611764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.612501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.614137 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 864/1692] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:07 2022] DELETE /identity/v3/projects/eb3ff1d4a6b541f7be2fa62fe0d686a5 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:07.618395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.620068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:07.688630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5c72e-965a-445c-aee1-548d19338517 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:07.696427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:07.697750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.698100 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.698642 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.699361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:07.701018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:07.720418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 829/1693] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:07 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:07.722712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.724505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:07.736354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e734007-cc65-4116-9ca8-5f8510da8633 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:07.741164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da47a71-3816-460d-83a7-6a972c1aed49 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:07.743231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:07.744107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.744437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.744878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] PATH_INFO: `/v3/users/26c36a11fd0442b5a0afd09dfe4396f4` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.746418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26c36a11fd0442b5a0afd09dfe4396f4)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:07.748485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:07.749812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ec11d0-a106-472e-bb6c-3ed23fb94357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:07.750628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ec11d0-a106-472e-bb6c-3ed23fb94357 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.750969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ec11d0-a106-472e-bb6c-3ed23fb94357 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.751188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ec11d0-a106-472e-bb6c-3ed23fb94357 admin admin] PATH_INFO: `/v3/projects/df97399ef8cf4da089964120403e4f8f/users/63659bff07cc466a8aa6019951f39965/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.756942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ec11d0-a106-472e-bb6c-3ed23fb94357 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df97399ef8cf4da089964120403e4f8f, user_id=63659bff07cc466a8aa6019951f39965, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:07.761120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ec11d0-a106-472e-bb6c-3ed23fb94357 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:07.775548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 830/1694] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:07 2022] PUT /identity/v3/projects/df97399ef8cf4da089964120403e4f8f/users/63659bff07cc466a8aa6019951f39965/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:07.787802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae4e8b0-4642-4a17-b722-9ca9ee3e2b8f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:07.788651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae4e8b0-4642-4a17-b722-9ca9ee3e2b8f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:07.788651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae4e8b0-4642-4a17-b722-9ca9ee3e2b8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:07.815072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26c36a11fd0442b5a0afd09dfe4396f4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.830531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26c36a11fd0442b5a0afd09dfe4396f4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.833984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3ae4e8b0-4642-4a17-b722-9ca9ee3e2b8f None None] MFA Rules not processed for user `63659bff07cc466a8aa6019951f39965`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:07.835543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3ae4e8b0-4642-4a17-b722-9ca9ee3e2b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:07.845529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26c36a11fd0442b5a0afd09dfe4396f4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.846316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26c36a11fd0442b5a0afd09dfe4396f4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.846924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26c36a11fd0442b5a0afd09dfe4396f4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:07.863069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:07.873594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 865/1695] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:07 2022] DELETE /identity/v3/users/26c36a11fd0442b5a0afd09dfe4396f4 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:07.943297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 831/1696] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:07 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:07.961723 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:07.964087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:08.064799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa05891-ea3b-46c4-b61c-336900ce2ef5 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:08.068980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba87f8c4-901b-42fd-99fa-fdd19df43bdb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:08.069607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87f8c4-901b-42fd-99fa-fdd19df43bdb service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:08.069977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87f8c4-901b-42fd-99fa-fdd19df43bdb service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:08.070233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87f8c4-901b-42fd-99fa-fdd19df43bdb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:08.168121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba87f8c4-901b-42fd-99fa-fdd19df43bdb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:08.169675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba87f8c4-901b-42fd-99fa-fdd19df43bdb service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:08.185536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 866/1697] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:07 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:08.293038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae4e8b0-4642-4a17-b722-9ca9ee3e2b8f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:08.294551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3ae4e8b0-4642-4a17-b722-9ca9ee3e2b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:08.352163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:08.412720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae4e8b0-4642-4a17-b722-9ca9ee3e2b8f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:08.495587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:08.496266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:08.496534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:08.496811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] PATH_INFO: `/v3/projects/4ae3d4f3769e4045b615b69517b75944` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:08.498526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ae3d4f3769e4045b615b69517b75944)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:08.500938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:08.570852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ae3d4f3769e4045b615b69517b75944', 'request_id': 'req-b7f65f9c-3ff3-4418-8e67-5d1961b26651'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:08.577978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ae3d4f3769e4045b615b69517b75944', 'request_id': 'req-b7f65f9c-3ff3-4418-8e67-5d1961b26651'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:08.585093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ae3d4f3769e4045b615b69517b75944', 'request_id': 'req-b7f65f9c-3ff3-4418-8e67-5d1961b26651'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:08.626889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] The token cache is being invalidate because project 4ae3d4f3769e4045b615b69517b75944 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:08.627515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:08.628597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b7f65f9c-3ff3-4418-8e67-5d1961b26651 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:08.630829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 832/1698] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:08 2022] DELETE /identity/v3/projects/4ae3d4f3769e4045b615b69517b75944 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:08.638024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc20d87c-84e3-4f8e-8faa-d2809a064c75 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:08.638645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc20d87c-84e3-4f8e-8faa-d2809a064c75 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:08.638956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc20d87c-84e3-4f8e-8faa-d2809a064c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:08.688662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-dc20d87c-84e3-4f8e-8faa-d2809a064c75 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:08.688662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-dc20d87c-84e3-4f8e-8faa-d2809a064c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:08.784701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 867/1699] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:08.792783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7287e27-ead8-4133-af92-d9f2df450fbb None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:08.793577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7287e27-ead8-4133-af92-d9f2df450fbb None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:08.793577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7287e27-ead8-4133-af92-d9f2df450fbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:08.824037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b7287e27-ead8-4133-af92-d9f2df450fbb None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:08.825341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b7287e27-ead8-4133-af92-d9f2df450fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:08.834698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 833/1700] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:08.839151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc20d87c-84e3-4f8e-8faa-d2809a064c75 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:08.851027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc20d87c-84e3-4f8e-8faa-d2809a064c75 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:08.856062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:08.856768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:08.857052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:08.857329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:08.858052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:08.858359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:08.860121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:08.877188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 868/1701] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:08.879936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7287e27-ead8-4133-af92-d9f2df450fbb None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:08.893491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7287e27-ead8-4133-af92-d9f2df450fbb None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:08.901188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989c4e1a-1908-4835-9fa6-54d536ebcd5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:08.901962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989c4e1a-1908-4835-9fa6-54d536ebcd5d admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:08.902489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989c4e1a-1908-4835-9fa6-54d536ebcd5d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:08.902625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989c4e1a-1908-4835-9fa6-54d536ebcd5d admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:08.903536 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989c4e1a-1908-4835-9fa6-54d536ebcd5d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:08.907276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989c4e1a-1908-4835-9fa6-54d536ebcd5d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:08.929970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:08.932047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:08.947106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d763880-e716-4e54-a8ef-e2f8537e7298 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:08.953189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:08.953419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 834/1702] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:08 2022] POST /identity/v3/projects => generated 385 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:08.953908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:08.954218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:08.954487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] PATH_INFO: `/v3/users/ec13ff540dd74041883081017fa6f49a` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:08.955627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec13ff540dd74041883081017fa6f49a)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:08.957494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989c4e1a-1908-4835-9fa6-54d536ebcd5d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:08.957618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:08.999558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:09.009411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:09.036729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec13ff540dd74041883081017fa6f49a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:09.056193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989c4e1a-1908-4835-9fa6-54d536ebcd5d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.061915 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.062823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.063201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.065178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.065871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.066902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec13ff540dd74041883081017fa6f49a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:09.068803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.091060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec13ff540dd74041883081017fa6f49a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:09.092970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec13ff540dd74041883081017fa6f49a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:09.093954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c68aad94-3aec-4a1e-be1e-bd02c0630505 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec13ff540dd74041883081017fa6f49a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:09.100164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 835/1703] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:08 2022] POST /identity/v3/users => generated 293 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:09.104563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.107305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:09.122461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 869/1704] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:08 2022] DELETE /identity/v3/users/ec13ff540dd74041883081017fa6f49a => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:09.128011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18806934-29b7-4321-8b37-d17cbfa62f3f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.128277 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18806934-29b7-4321-8b37-d17cbfa62f3f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.129073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18806934-29b7-4321-8b37-d17cbfa62f3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.142030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f26538-ef6e-4cfd-be79-e5469af36f42 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.160709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-18806934-29b7-4321-8b37-d17cbfa62f3f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:09.165173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-18806934-29b7-4321-8b37-d17cbfa62f3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:09.250420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cd5e47-93eb-4102-a3c0-e099fff01025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.251055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd5e47-93eb-4102-a3c0-e099fff01025 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.251326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd5e47-93eb-4102-a3c0-e099fff01025 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.251604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd5e47-93eb-4102-a3c0-e099fff01025 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.252332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cd5e47-93eb-4102-a3c0-e099fff01025 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.254027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cd5e47-93eb-4102-a3c0-e099fff01025 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.267048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 836/1705] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:09 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:09.273175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cd5e47-93eb-4102-a3c0-e099fff01025 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.276196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 870/1706] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:09.286843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18806934-29b7-4321-8b37-d17cbfa62f3f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.288447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-18806934-29b7-4321-8b37-d17cbfa62f3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:09.293279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cd5e47-93eb-4102-a3c0-e099fff01025 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.307970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5700be77-140a-4b97-83ff-fe4fe8cc5ac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.308752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5700be77-140a-4b97-83ff-fe4fe8cc5ac4 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.309039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5700be77-140a-4b97-83ff-fe4fe8cc5ac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.309375 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5700be77-140a-4b97-83ff-fe4fe8cc5ac4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.310023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5700be77-140a-4b97-83ff-fe4fe8cc5ac4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.311766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5700be77-140a-4b97-83ff-fe4fe8cc5ac4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.327241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 837/1707] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:09 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:09.328368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5700be77-140a-4b97-83ff-fe4fe8cc5ac4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.346650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5700be77-140a-4b97-83ff-fe4fe8cc5ac4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.354213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840ce5ec-2182-4d8f-93c7-ee39a4c0664d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.355227 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ce5ec-2182-4d8f-93c7-ee39a4c0664d admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.355609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ce5ec-2182-4d8f-93c7-ee39a4c0664d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.355959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ce5ec-2182-4d8f-93c7-ee39a4c0664d admin admin] PATH_INFO: `/v3/projects/19a9324931d249f0a0b46e3ecc6f405a/users/56f2c1748f4d454896d84249dab50796/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.378627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18806934-29b7-4321-8b37-d17cbfa62f3f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.382340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ce5ec-2182-4d8f-93c7-ee39a4c0664d admin admin] RBAC: Authorizing `identity:create_grant(project_id=19a9324931d249f0a0b46e3ecc6f405a, user_id=56f2c1748f4d454896d84249dab50796, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.383841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d630b81-2ba1-422d-9dc6-3728b1ac3043 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.384581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d630b81-2ba1-422d-9dc6-3728b1ac3043 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.384854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d630b81-2ba1-422d-9dc6-3728b1ac3043 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.385125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ce5ec-2182-4d8f-93c7-ee39a4c0664d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.385289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d630b81-2ba1-422d-9dc6-3728b1ac3043 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.396662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d630b81-2ba1-422d-9dc6-3728b1ac3043 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.397990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d630b81-2ba1-422d-9dc6-3728b1ac3043 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.404198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 838/1708] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:09 2022] PUT /identity/v3/projects/19a9324931d249f0a0b46e3ecc6f405a/users/56f2c1748f4d454896d84249dab50796/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:09.413796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f33ae-0244-47a5-887c-38ca5a45d58f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.414131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f33ae-0244-47a5-887c-38ca5a45d58f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.414440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f33ae-0244-47a5-887c-38ca5a45d58f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.416438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 871/1709] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:09.448201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c95f33ae-0244-47a5-887c-38ca5a45d58f None None] MFA Rules not processed for user `56f2c1748f4d454896d84249dab50796`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:09.449586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c95f33ae-0244-47a5-887c-38ca5a45d58f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:09.549298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 839/1710] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:09 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:09.553732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d630b81-2ba1-422d-9dc6-3728b1ac3043 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.565043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d630b81-2ba1-422d-9dc6-3728b1ac3043 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.619112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:09.633388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.634929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.635302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.635482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.636339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.641602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.678202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 872/1711] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:09 2022] POST /identity/v3/projects => generated 387 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:09.681963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95f33ae-0244-47a5-887c-38ca5a45d58f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.690778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:09.699427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95f33ae-0244-47a5-887c-38ca5a45d58f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.750867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.750867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:09.799683 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aad3c8a-ab50-4642-91ae-315713f6891d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.800319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad3c8a-ab50-4642-91ae-315713f6891d admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.800606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad3c8a-ab50-4642-91ae-315713f6891d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.800880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad3c8a-ab50-4642-91ae-315713f6891d admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.801679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aad3c8a-ab50-4642-91ae-315713f6891d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.804957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aad3c8a-ab50-4642-91ae-315713f6891d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.840980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 840/1712] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:09 2022] POST /identity/v3/users => generated 300 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:09.845305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aad3c8a-ab50-4642-91ae-315713f6891d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.863934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aad3c8a-ab50-4642-91ae-315713f6891d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.867780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94aadc3-b6e4-42c5-8e71-7b6236106e9f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.874402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d585c1-384d-4d5a-ac03-449d562f59ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.875053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d585c1-384d-4d5a-ac03-449d562f59ba admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.875386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d585c1-384d-4d5a-ac03-449d562f59ba admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.875667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d585c1-384d-4d5a-ac03-449d562f59ba admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.875793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.876378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d585c1-384d-4d5a-ac03-449d562f59ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.876518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.877003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.877287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] PATH_INFO: `/v3/projects/18b37d8b6ce64b7cbae756808a0caa5d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.878065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d585c1-384d-4d5a-ac03-449d562f59ba admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.878680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] RBAC: Authorizing `identity:delete_project(project_id=18b37d8b6ce64b7cbae756808a0caa5d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.881112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.894831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 841/1713] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:09 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:09.898944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d585c1-384d-4d5a-ac03-449d562f59ba admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:09.925540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d585c1-384d-4d5a-ac03-449d562f59ba admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:09.934250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac06166-7df9-4004-933c-0289f257b4a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:09.935055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac06166-7df9-4004-933c-0289f257b4a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:09.935335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac06166-7df9-4004-933c-0289f257b4a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:09.935659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac06166-7df9-4004-933c-0289f257b4a7 admin admin] PATH_INFO: `/v3/projects/f7a0bfcf32644a9298389d032aac236d/users/a87d5dad9c2a4d63b895e8f69864cee9/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:09.960847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18b37d8b6ce64b7cbae756808a0caa5d', 'request_id': 'req-1d48829b-dda9-4b70-8c98-8235b2d0b768'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:09.968029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac06166-7df9-4004-933c-0289f257b4a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7a0bfcf32644a9298389d032aac236d, user_id=a87d5dad9c2a4d63b895e8f69864cee9, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:09.970646 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18b37d8b6ce64b7cbae756808a0caa5d', 'request_id': 'req-1d48829b-dda9-4b70-8c98-8235b2d0b768'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:09.971636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac06166-7df9-4004-933c-0289f257b4a7 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:09.997324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18b37d8b6ce64b7cbae756808a0caa5d', 'request_id': 'req-1d48829b-dda9-4b70-8c98-8235b2d0b768'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.012231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 842/1714] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:09 2022] PUT /identity/v3/projects/f7a0bfcf32644a9298389d032aac236d/users/a87d5dad9c2a4d63b895e8f69864cee9/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:10.015361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac06166-7df9-4004-933c-0289f257b4a7 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:10.023969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] The token cache is being invalidate because project 18b37d8b6ce64b7cbae756808a0caa5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:10.024375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.027524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.030444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 873/1715] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:09 2022] DELETE /identity/v3/projects/18b37d8b6ce64b7cbae756808a0caa5d => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:10.034257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:10.035723 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:10.118908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac06166-7df9-4004-933c-0289f257b4a7 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:10.126366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:10.126984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.127264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.127533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.128275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:10.130018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:10.132724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d48829b-dda9-4b70-8c98-8235b2d0b768 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:10.138147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:10.138729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.140054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.140054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] PATH_INFO: `/v3/users/63659bff07cc466a8aa6019951f39965` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.141829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 843/1716] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:10 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:10.141829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:10.143772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:10.145921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] RBAC: Authorizing `identity:delete_user(user_id=63659bff07cc466a8aa6019951f39965)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:10.147880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:10.157306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17a3f10-df0c-4635-bc2f-ecfc08bb4e88 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:10.157685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:10.162340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:10.162989 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.163248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.163538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] PATH_INFO: `/v3/users/6991a40984164029a4574c57fe3008f1` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.165198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6991a40984164029a4574c57fe3008f1)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:10.167554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:10.213069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63659bff07cc466a8aa6019951f39965'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.221638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63659bff07cc466a8aa6019951f39965'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.230850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6991a40984164029a4574c57fe3008f1'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.236055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63659bff07cc466a8aa6019951f39965'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.236834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:10.237010 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63659bff07cc466a8aa6019951f39965'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.237585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63659bff07cc466a8aa6019951f39965'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.250238 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6991a40984164029a4574c57fe3008f1'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.261874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 874/1717] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:10 2022] DELETE /identity/v3/users/63659bff07cc466a8aa6019951f39965 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:10.261874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:10.261874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:10.315558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6991a40984164029a4574c57fe3008f1'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.316387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6991a40984164029a4574c57fe3008f1'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.317232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-753d3213-1dc9-44a1-aa55-3a1cab131305 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6991a40984164029a4574c57fe3008f1'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.339874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 844/1718] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:10 2022] DELETE /identity/v3/users/6991a40984164029a4574c57fe3008f1 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:10.341628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1429c9c7-9791-4ead-9670-158bb6c3fbfc None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.341954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1429c9c7-9791-4ead-9670-158bb6c3fbfc None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.344524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1429c9c7-9791-4ead-9670-158bb6c3fbfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.376951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d9a2b5-2d10-435f-b160-621b138d1894 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:10.377511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-1429c9c7-9791-4ead-9670-158bb6c3fbfc None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:10.379074 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1429c9c7-9791-4ead-9670-158bb6c3fbfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:10.472940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 845/1719] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:10.483333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4430a7-a2bb-461b-983f-538a580460dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:10.484576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4430a7-a2bb-461b-983f-538a580460dd admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.484936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4430a7-a2bb-461b-983f-538a580460dd admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.485215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4430a7-a2bb-461b-983f-538a580460dd admin admin] PATH_INFO: `/v3/projects/f7a0bfcf32644a9298389d032aac236d/users/a87d5dad9c2a4d63b895e8f69864cee9/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.487200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1429c9c7-9791-4ead-9670-158bb6c3fbfc None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:10.488117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4430a7-a2bb-461b-983f-538a580460dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7a0bfcf32644a9298389d032aac236d, user_id=a87d5dad9c2a4d63b895e8f69864cee9, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:10.488964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1429c9c7-9791-4ead-9670-158bb6c3fbfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:10.491254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4430a7-a2bb-461b-983f-538a580460dd admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:10.510921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 875/1720] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:10 2022] PUT /identity/v3/projects/f7a0bfcf32644a9298389d032aac236d/users/a87d5dad9c2a4d63b895e8f69864cee9/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:10.522127 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb8761-12fd-4ada-af66-9a4e68cac5b8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.522408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb8761-12fd-4ada-af66-9a4e68cac5b8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.522702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb8761-12fd-4ada-af66-9a4e68cac5b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.559765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-2efb8761-12fd-4ada-af66-9a4e68cac5b8 None None] MFA Rules not processed for user `a87d5dad9c2a4d63b895e8f69864cee9`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:10.562045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2efb8761-12fd-4ada-af66-9a4e68cac5b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:10.572364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1429c9c7-9791-4ead-9670-158bb6c3fbfc None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:10.588835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:10.679869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:10.680525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.680826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.681096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.729827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 876/1721] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:10 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:10.748123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb8761-12fd-4ada-af66-9a4e68cac5b8 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:10.751132 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2efb8761-12fd-4ada-af66-9a4e68cac5b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:10.771297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:10.815627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:10.817472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:10.853789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 846/1722] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:10.855138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:10.858939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:10.864392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb8761-12fd-4ada-af66-9a4e68cac5b8 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:10.870718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58faae7-0250-4e20-bf1c-c25a8b401b75 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:10.872257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58faae7-0250-4e20-bf1c-c25a8b401b75 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.872736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58faae7-0250-4e20-bf1c-c25a8b401b75 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.873155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58faae7-0250-4e20-bf1c-c25a8b401b75 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.875251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9800f533-b01e-44ca-9f4f-7d25eb2ee974 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:10.880419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:10.881064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:10.881348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:10.881673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] PATH_INFO: `/v3/projects/df97399ef8cf4da089964120403e4f8f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:10.882792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] RBAC: Authorizing `identity:delete_project(project_id=df97399ef8cf4da089964120403e4f8f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:10.886598 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:10.898419 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58faae7-0250-4e20-bf1c-c25a8b401b75 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:10.901387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58faae7-0250-4e20-bf1c-c25a8b401b75 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:10.932502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 877/1723] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:10 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:10.959221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df97399ef8cf4da089964120403e4f8f', 'request_id': 'req-3baa7ba2-6a10-4551-a880-17d873e8886a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.967483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df97399ef8cf4da089964120403e4f8f', 'request_id': 'req-3baa7ba2-6a10-4551-a880-17d873e8886a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.976107 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df97399ef8cf4da089964120403e4f8f', 'request_id': 'req-3baa7ba2-6a10-4551-a880-17d873e8886a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.997952 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] The token cache is being invalidate because project df97399ef8cf4da089964120403e4f8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:10.998567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:10.999844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3baa7ba2-6a10-4551-a880-17d873e8886a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.003874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 847/1724] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:10 2022] DELETE /identity/v3/projects/df97399ef8cf4da089964120403e4f8f => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:11.009186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdccb17d-94be-4b4e-8f1e-ddab88a288a6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.009469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdccb17d-94be-4b4e-8f1e-ddab88a288a6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.009739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdccb17d-94be-4b4e-8f1e-ddab88a288a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.036008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-cdccb17d-94be-4b4e-8f1e-ddab88a288a6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:11.037602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-cdccb17d-94be-4b4e-8f1e-ddab88a288a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:11.168926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 878/1725] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:11.180504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c91049-dc9c-4a41-8cc4-34da653ec48e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.180750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c91049-dc9c-4a41-8cc4-34da653ec48e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.181051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c91049-dc9c-4a41-8cc4-34da653ec48e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.212319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a3c91049-dc9c-4a41-8cc4-34da653ec48e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:11.214405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a3c91049-dc9c-4a41-8cc4-34da653ec48e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:11.229849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 848/1726] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:11.235538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdccb17d-94be-4b4e-8f1e-ddab88a288a6 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.250672 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdccb17d-94be-4b4e-8f1e-ddab88a288a6 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:11.257990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:11.258729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.259070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.259426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.260670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:11.261059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:11.263002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:11.289227 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c91049-dc9c-4a41-8cc4-34da653ec48e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.291763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a3c91049-dc9c-4a41-8cc4-34da653ec48e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:11.295979 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 879/1727] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:11.300550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.321440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c91049-dc9c-4a41-8cc4-34da653ec48e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:11.327634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca50cb0-d5ce-4a47-a856-d90621ffb13c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:11.334012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:11.334012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.334012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.334299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] PATH_INFO: `/v3/projects/bfd77a0fc47a4e6fac66f7f9ccad631f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.335470 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfd77a0fc47a4e6fac66f7f9ccad631f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:11.337528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:11.338212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bd7eab-8167-4de1-ad89-ecd3767ba747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:11.338841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bd7eab-8167-4de1-ad89-ecd3767ba747 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.339122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bd7eab-8167-4de1-ad89-ecd3767ba747 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.339395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bd7eab-8167-4de1-ad89-ecd3767ba747 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.340041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bd7eab-8167-4de1-ad89-ecd3767ba747 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:11.343531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bd7eab-8167-4de1-ad89-ecd3767ba747 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:11.398795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 880/1728] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:11 2022] POST /identity/v3/projects => generated 371 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:11.404044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bd7eab-8167-4de1-ad89-ecd3767ba747 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.442089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfd77a0fc47a4e6fac66f7f9ccad631f', 'request_id': 'req-7e472c66-2166-4ac9-bfaa-82e23643acdd'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.448517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:11.456757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfd77a0fc47a4e6fac66f7f9ccad631f', 'request_id': 'req-7e472c66-2166-4ac9-bfaa-82e23643acdd'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.469873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfd77a0fc47a4e6fac66f7f9ccad631f', 'request_id': 'req-7e472c66-2166-4ac9-bfaa-82e23643acdd'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.492755 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] The token cache is being invalidate because project bfd77a0fc47a4e6fac66f7f9ccad631f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:11.493307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.494785 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.497312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 849/1729] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:11 2022] DELETE /identity/v3/projects/bfd77a0fc47a4e6fac66f7f9ccad631f => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:11.503332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.507549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:11.519290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bd7eab-8167-4de1-ad89-ecd3767ba747 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:11.526881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:11.527502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.527783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.528057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.528772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:11.530883 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:11.581278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:11.585362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 881/1730] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:11 2022] POST /identity/v3/users => generated 286 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:11.590048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.593116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:11.620327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5129c4-fc8e-403a-9160-9c6a9b94eb74 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:11.631011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c32c9c-76d9-4024-93e9-638d11bff6ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:11.631644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c32c9c-76d9-4024-93e9-638d11bff6ff admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.631924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c32c9c-76d9-4024-93e9-638d11bff6ff admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.632541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c32c9c-76d9-4024-93e9-638d11bff6ff admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.633677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c32c9c-76d9-4024-93e9-638d11bff6ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:11.635359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c32c9c-76d9-4024-93e9-638d11bff6ff admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:11.655932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 882/1731] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:11 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:11.657557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c32c9c-76d9-4024-93e9-638d11bff6ff admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.659629 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e472c66-2166-4ac9-bfaa-82e23643acdd admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:11.668897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:11.669806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.671121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.671701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] PATH_INFO: `/v3/users/53285422cfa54c30814540887f3d710a` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.673712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c32c9c-76d9-4024-93e9-638d11bff6ff admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:11.674251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=53285422cfa54c30814540887f3d710a)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:11.677816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:11.688925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74de080b-b6ac-4f2d-b17c-cb2aa052a81f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:11.691119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74de080b-b6ac-4f2d-b17c-cb2aa052a81f admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.691977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74de080b-b6ac-4f2d-b17c-cb2aa052a81f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.692702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74de080b-b6ac-4f2d-b17c-cb2aa052a81f admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.694028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74de080b-b6ac-4f2d-b17c-cb2aa052a81f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:11.697172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74de080b-b6ac-4f2d-b17c-cb2aa052a81f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:11.723651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 883/1732] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:11 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:11.728276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74de080b-b6ac-4f2d-b17c-cb2aa052a81f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.754800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74de080b-b6ac-4f2d-b17c-cb2aa052a81f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:11.765311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d600bc13-aa3e-4554-9a9a-8b2eaed92384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:11.766706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600bc13-aa3e-4554-9a9a-8b2eaed92384 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.767460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600bc13-aa3e-4554-9a9a-8b2eaed92384 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.768198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600bc13-aa3e-4554-9a9a-8b2eaed92384 admin admin] PATH_INFO: `/v3/projects/69f8ba90c84748bda85edc2c7dcd4f73/users/32d8eb0025ec43ce8d967ece7d2f0dd3/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.798887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d600bc13-aa3e-4554-9a9a-8b2eaed92384 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69f8ba90c84748bda85edc2c7dcd4f73, user_id=32d8eb0025ec43ce8d967ece7d2f0dd3, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:11.804811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d600bc13-aa3e-4554-9a9a-8b2eaed92384 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:11.805074 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53285422cfa54c30814540887f3d710a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.834880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 884/1733] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:11 2022] PUT /identity/v3/projects/69f8ba90c84748bda85edc2c7dcd4f73/users/32d8eb0025ec43ce8d967ece7d2f0dd3/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:11.840732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619011b9-be3a-4141-9c0d-ff2757f6c900 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:11.841346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53285422cfa54c30814540887f3d710a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.841526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619011b9-be3a-4141-9c0d-ff2757f6c900 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:11.841863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619011b9-be3a-4141-9c0d-ff2757f6c900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:11.886877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-619011b9-be3a-4141-9c0d-ff2757f6c900 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:11.888452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-619011b9-be3a-4141-9c0d-ff2757f6c900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:11.900652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53285422cfa54c30814540887f3d710a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.901956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53285422cfa54c30814540887f3d710a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.905684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53285422cfa54c30814540887f3d710a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:11.944787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 850/1734] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:11 2022] DELETE /identity/v3/users/53285422cfa54c30814540887f3d710a => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:11.946678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:11.959769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:12.009424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 885/1735] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:12.026874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619011b9-be3a-4141-9c0d-ff2757f6c900 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:12.028354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-619011b9-be3a-4141-9c0d-ff2757f6c900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:12.069263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf732eb4-1d33-4c85-a725-0cc86aa1a1f2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:12.085396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:12.086593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.087097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.087569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] PATH_INFO: `/v3/users/56f2c1748f4d454896d84249dab50796` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.090565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] RBAC: Authorizing `identity:delete_user(user_id=56f2c1748f4d454896d84249dab50796)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:12.095769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:12.204606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619011b9-be3a-4141-9c0d-ff2757f6c900 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:12.207950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56f2c1748f4d454896d84249dab50796'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.216463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:12.229306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56f2c1748f4d454896d84249dab50796'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.252009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56f2c1748f4d454896d84249dab50796'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.252850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56f2c1748f4d454896d84249dab50796'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.254233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d1c6afb-3afa-4799-a842-3bdfb8c6023c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56f2c1748f4d454896d84249dab50796'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.255555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:12.306799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 851/1736] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:11 2022] DELETE /identity/v3/users/56f2c1748f4d454896d84249dab50796 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:12.314700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbc1ec9-55ff-428b-a09b-3790cb232d3f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.315278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbc1ec9-55ff-428b-a09b-3790cb232d3f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.316805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbc1ec9-55ff-428b-a09b-3790cb232d3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.337373 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:12.338157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.338552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.338838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.341295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:12.356253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:12.359099 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:12.371770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 886/1737] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:12.376701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-bfbc1ec9-55ff-428b-a09b-3790cb232d3f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:12.379086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bfbc1ec9-55ff-428b-a09b-3790cb232d3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:12.462321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 852/1738] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:12.476989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:12.480831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:12.484484 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbc1ec9-55ff-428b-a09b-3790cb232d3f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:12.486097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bfbc1ec9-55ff-428b-a09b-3790cb232d3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:12.506805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbc1ec9-55ff-428b-a09b-3790cb232d3f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:12.507324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70f0c6f-0387-4701-8739-6da90c256c5a service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:12.523970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:12.525149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.526089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.526367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] PATH_INFO: `/v3/projects/d0018036835e4b0bac9de1eccd34dd22` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.528252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0018036835e4b0bac9de1eccd34dd22)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:12.530805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:12.595811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55735f8d-2861-4161-b58c-cee87b8987dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:12.596486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55735f8d-2861-4161-b58c-cee87b8987dc service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.596858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55735f8d-2861-4161-b58c-cee87b8987dc service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.598030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55735f8d-2861-4161-b58c-cee87b8987dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.606213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0018036835e4b0bac9de1eccd34dd22', 'request_id': 'req-19315cd2-f611-46c9-9c04-f2d9023022b9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.610047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55735f8d-2861-4161-b58c-cee87b8987dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:12.612110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55735f8d-2861-4161-b58c-cee87b8987dc service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:12.615878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0018036835e4b0bac9de1eccd34dd22', 'request_id': 'req-19315cd2-f611-46c9-9c04-f2d9023022b9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.626550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 887/1739] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:12.628581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0018036835e4b0bac9de1eccd34dd22', 'request_id': 'req-19315cd2-f611-46c9-9c04-f2d9023022b9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.648330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] The token cache is being invalidate because project d0018036835e4b0bac9de1eccd34dd22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:12.648631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.649531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-19315cd2-f611-46c9-9c04-f2d9023022b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:12.655749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 853/1740] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:12 2022] DELETE /identity/v3/projects/d0018036835e4b0bac9de1eccd34dd22 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:12.657917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f5817e-be94-4a93-a0d0-c6d1f8f9f76a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.658436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f5817e-be94-4a93-a0d0-c6d1f8f9f76a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.658897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f5817e-be94-4a93-a0d0-c6d1f8f9f76a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.694441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a3f5817e-be94-4a93-a0d0-c6d1f8f9f76a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:12.695800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a3f5817e-be94-4a93-a0d0-c6d1f8f9f76a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:12.784076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 888/1741] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:12.792692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf23ace3-aa92-4f43-9ddb-50bbd48a238b None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.792692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf23ace3-aa92-4f43-9ddb-50bbd48a238b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.792692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf23ace3-aa92-4f43-9ddb-50bbd48a238b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.816620 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-cf23ace3-aa92-4f43-9ddb-50bbd48a238b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:12.817554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cf23ace3-aa92-4f43-9ddb-50bbd48a238b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:12.837826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f5817e-be94-4a93-a0d0-c6d1f8f9f76a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:12.847632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 854/1742] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:12.853318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f5817e-be94-4a93-a0d0-c6d1f8f9f76a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:12.860079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:12.860912 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.861197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.861530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.862240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:12.862542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:12.864261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:12.883869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 889/1743] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:52:12.888389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf23ace3-aa92-4f43-9ddb-50bbd48a238b None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:12.899234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf23ace3-aa92-4f43-9ddb-50bbd48a238b None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:12.904164 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:12.905856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:12.905856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:12.906038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:12.906696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:12.908695 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:12.947214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 855/1744] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:12 2022] POST /identity/v3/projects => generated 365 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:12.950073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:12.963177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ea0914-7407-481e-800b-3b1439f46c08 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:13.004693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:13.007173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:13.056431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:13.072164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b837feeb-9084-45c7-9371-073b3d46e2ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:13.072843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b837feeb-9084-45c7-9371-073b3d46e2ce admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.073123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b837feeb-9084-45c7-9371-073b3d46e2ce admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.073450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b837feeb-9084-45c7-9371-073b3d46e2ce admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.074273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b837feeb-9084-45c7-9371-073b3d46e2ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:13.077892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b837feeb-9084-45c7-9371-073b3d46e2ce admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:13.083113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:13.108679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 890/1745] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:12 2022] POST /identity/v3/users => generated 283 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:13.113346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b837feeb-9084-45c7-9371-073b3d46e2ce admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:13.133195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b837feeb-9084-45c7-9371-073b3d46e2ce admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:13.139768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf3d5ad-76c4-4c17-a538-0724192e7bee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:13.141729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf3d5ad-76c4-4c17-a538-0724192e7bee admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.142094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf3d5ad-76c4-4c17-a538-0724192e7bee admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.142499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf3d5ad-76c4-4c17-a538-0724192e7bee admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.143345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf3d5ad-76c4-4c17-a538-0724192e7bee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:13.145577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf3d5ad-76c4-4c17-a538-0724192e7bee admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:13.160773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 891/1746] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:13 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:13.165564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf3d5ad-76c4-4c17-a538-0724192e7bee admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:13.168469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4e396-9e43-4f6f-8173-e548ce9f4c67 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:13.185052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:13.187990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.189287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.189746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] PATH_INFO: `/v3/projects/19a9324931d249f0a0b46e3ecc6f405a` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.192008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] RBAC: Authorizing `identity:delete_project(project_id=19a9324931d249f0a0b46e3ecc6f405a)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:13.193342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf3d5ad-76c4-4c17-a538-0724192e7bee admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:13.199883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:13.207736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41715ca-c542-4380-9c14-afda25b184c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:13.208438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41715ca-c542-4380-9c14-afda25b184c4 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.209428 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41715ca-c542-4380-9c14-afda25b184c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.209668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41715ca-c542-4380-9c14-afda25b184c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.210753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41715ca-c542-4380-9c14-afda25b184c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:13.212020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41715ca-c542-4380-9c14-afda25b184c4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:13.238272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 892/1747] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:13 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:13.239742 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41715ca-c542-4380-9c14-afda25b184c4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:13.253476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41715ca-c542-4380-9c14-afda25b184c4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:13.262534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8839fc52-59b7-4610-989d-0a08c9b309ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:13.263689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8839fc52-59b7-4610-989d-0a08c9b309ec admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.264402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8839fc52-59b7-4610-989d-0a08c9b309ec admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.265032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8839fc52-59b7-4610-989d-0a08c9b309ec admin admin] PATH_INFO: `/v3/projects/abc092e5568b4a3cb5179c274198fd60/users/59da7a35ed8043c49a1482dc94a1e83b/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.285880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8839fc52-59b7-4610-989d-0a08c9b309ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=abc092e5568b4a3cb5179c274198fd60, user_id=59da7a35ed8043c49a1482dc94a1e83b, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:13.289744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8839fc52-59b7-4610-989d-0a08c9b309ec admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:13.309405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19a9324931d249f0a0b46e3ecc6f405a', 'request_id': 'req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.316604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 893/1748] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:13 2022] PUT /identity/v3/projects/abc092e5568b4a3cb5179c274198fd60/users/59da7a35ed8043c49a1482dc94a1e83b/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:13.328799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2d9c8-a12c-4722-b67b-26ca5bf5556e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.328799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2d9c8-a12c-4722-b67b-26ca5bf5556e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.328799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2d9c8-a12c-4722-b67b-26ca5bf5556e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.333742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19a9324931d249f0a0b46e3ecc6f405a', 'request_id': 'req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.372400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-aee2d9c8-a12c-4722-b67b-26ca5bf5556e None None] MFA Rules not processed for user `59da7a35ed8043c49a1482dc94a1e83b`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:13.380702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-aee2d9c8-a12c-4722-b67b-26ca5bf5556e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:13.389017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19a9324931d249f0a0b46e3ecc6f405a', 'request_id': 'req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.413993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] The token cache is being invalidate because project 19a9324931d249f0a0b46e3ecc6f405a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:13.414371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.416750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.421553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 856/1749] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:13 2022] DELETE /identity/v3/projects/19a9324931d249f0a0b46e3ecc6f405a => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:13.425351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:13.428750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:13.550993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dd27fe-a8ba-43eb-8101-8cc5a2746aff admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:13.602539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 894/1750] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:13 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:13.608649 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee2d9c8-a12c-4722-b67b-26ca5bf5556e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:13.611653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-aee2d9c8-a12c-4722-b67b-26ca5bf5556e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:13.676977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:13.678548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.679194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.680940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] PATH_INFO: `/v3/users/a87d5dad9c2a4d63b895e8f69864cee9` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.683444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a87d5dad9c2a4d63b895e8f69864cee9)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:13.687326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:13.698478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:13.768601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a87d5dad9c2a4d63b895e8f69864cee9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.773412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee2d9c8-a12c-4722-b67b-26ca5bf5556e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:13.779552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a87d5dad9c2a4d63b895e8f69864cee9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.780450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4622f657-2d41-43f8-950f-708ca5ed2c8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:13.781655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4622f657-2d41-43f8-950f-708ca5ed2c8a admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.781976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4622f657-2d41-43f8-950f-708ca5ed2c8a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.782410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4622f657-2d41-43f8-950f-708ca5ed2c8a admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.783278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4622f657-2d41-43f8-950f-708ca5ed2c8a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:13.786278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4622f657-2d41-43f8-950f-708ca5ed2c8a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:13.798692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a87d5dad9c2a4d63b895e8f69864cee9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.799738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a87d5dad9c2a4d63b895e8f69864cee9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.800471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a87d5dad9c2a4d63b895e8f69864cee9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:13.825992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 857/1751] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:13 2022] DELETE /identity/v3/users/a87d5dad9c2a4d63b895e8f69864cee9 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:13.832318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 895/1752] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:13 2022] POST /identity/v3/projects => generated 367 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:13.834172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:13.837030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:13.949791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4223c1-40ca-4c57-9917-dbe35cdea5d3 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:13.950175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4223c1-40ca-4c57-9917-dbe35cdea5d3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:13.950888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4223c1-40ca-4c57-9917-dbe35cdea5d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:13.958186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09ad18-48fe-4b9c-affd-27ed4fdef2b3 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:13.997106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:14.016663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0b4223c1-40ca-4c57-9917-dbe35cdea5d3 None None] MFA Rules not processed for user `32d8eb0025ec43ce8d967ece7d2f0dd3`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:14.018139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0b4223c1-40ca-4c57-9917-dbe35cdea5d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:14.085353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c226c18-3722-4e22-b055-f7089ba5d6de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.086081 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c226c18-3722-4e22-b055-f7089ba5d6de admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.086426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c226c18-3722-4e22-b055-f7089ba5d6de admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.086764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c226c18-3722-4e22-b055-f7089ba5d6de admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.087516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c226c18-3722-4e22-b055-f7089ba5d6de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.089670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c226c18-3722-4e22-b055-f7089ba5d6de admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.118604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 858/1753] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:13 2022] POST /identity/v3/users => generated 290 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:14.124082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c226c18-3722-4e22-b055-f7089ba5d6de admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.138305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c226c18-3722-4e22-b055-f7089ba5d6de admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.147712 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fe9e9c-9423-4d88-9837-8272e7b1f0d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.148714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fe9e9c-9423-4d88-9837-8272e7b1f0d3 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.151597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fe9e9c-9423-4d88-9837-8272e7b1f0d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.152118 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fe9e9c-9423-4d88-9837-8272e7b1f0d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.153017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fe9e9c-9423-4d88-9837-8272e7b1f0d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.158105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fe9e9c-9423-4d88-9837-8272e7b1f0d3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.188998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 859/1754] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:14 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:14.191234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fe9e9c-9423-4d88-9837-8272e7b1f0d3 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.212894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 896/1755] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:13 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:14.213893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4223c1-40ca-4c57-9917-dbe35cdea5d3 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.216026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0b4223c1-40ca-4c57-9917-dbe35cdea5d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:14.221554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fe9e9c-9423-4d88-9837-8272e7b1f0d3 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.235163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5272e4-37ee-4af3-b3b1-9126eb91d777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.235774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5272e4-37ee-4af3-b3b1-9126eb91d777 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.236055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5272e4-37ee-4af3-b3b1-9126eb91d777 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.236414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5272e4-37ee-4af3-b3b1-9126eb91d777 admin admin] PATH_INFO: `/v3/projects/b5ae047cf98d43779955e10358da705d/users/4d59ac7f5f6447519d8ec204ab414d5e/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.238685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4223c1-40ca-4c57-9917-dbe35cdea5d3 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.243875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:14.253268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.253802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.254563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.254563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.255259 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.257158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.272382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5272e4-37ee-4af3-b3b1-9126eb91d777 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5ae047cf98d43779955e10358da705d, user_id=4d59ac7f5f6447519d8ec204ab414d5e, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.274928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5272e4-37ee-4af3-b3b1-9126eb91d777 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.304855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 897/1756] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:14 2022] POST /identity/v3/projects => generated 371 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:14.305838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.313113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:14.313328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 860/1757] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:14 2022] PUT /identity/v3/projects/b5ae047cf98d43779955e10358da705d/users/4d59ac7f5f6447519d8ec204ab414d5e/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:14.314237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5272e4-37ee-4af3-b3b1-9126eb91d777 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.390721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805cee2b-d10c-45bd-8b4b-237f6fcb04c1 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.395492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.396431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.396431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.396668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] PATH_INFO: `/v3/projects/f7a0bfcf32644a9298389d032aac236d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.397793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7a0bfcf32644a9298389d032aac236d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.399823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.412180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5272e4-37ee-4af3-b3b1-9126eb91d777 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.417439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187e12bf-0016-4c9d-9b42-6077970549bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.418202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187e12bf-0016-4c9d-9b42-6077970549bd admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.418580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187e12bf-0016-4c9d-9b42-6077970549bd admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.418922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187e12bf-0016-4c9d-9b42-6077970549bd admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.419651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187e12bf-0016-4c9d-9b42-6077970549bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.422995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187e12bf-0016-4c9d-9b42-6077970549bd admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.450154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 861/1758] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:14 2022] POST /identity/v3/users => generated 292 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:14.451533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187e12bf-0016-4c9d-9b42-6077970549bd admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.456501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7a0bfcf32644a9298389d032aac236d', 'request_id': 'req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:14.463989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7a0bfcf32644a9298389d032aac236d', 'request_id': 'req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:14.464890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187e12bf-0016-4c9d-9b42-6077970549bd admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.474698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7a0bfcf32644a9298389d032aac236d', 'request_id': 'req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:14.487838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] The token cache is being invalidate because project f7a0bfcf32644a9298389d032aac236d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:14.488235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:14.488984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:14.494157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 898/1759] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:14 2022] DELETE /identity/v3/projects/f7a0bfcf32644a9298389d032aac236d => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:14.494157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.494157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:14.505842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:14.563504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2407f211-43d1-4a7c-9a52-5d706b39e21f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.565032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407f211-43d1-4a7c-9a52-5d706b39e21f admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.565402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407f211-43d1-4a7c-9a52-5d706b39e21f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.565832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407f211-43d1-4a7c-9a52-5d706b39e21f admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.566951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2407f211-43d1-4a7c-9a52-5d706b39e21f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.569446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2407f211-43d1-4a7c-9a52-5d706b39e21f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.581740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ab2937-5557-41dc-89ea-f0e9ca5570d1 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.587201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 862/1760] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:14 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:14.588609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.589113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd954065-b26b-45a4-b052-757536ea9142 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.589245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.589494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.589689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd954065-b26b-45a4-b052-757536ea9142 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.589812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.590131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd954065-b26b-45a4-b052-757536ea9142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.590430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.592226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.608442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 899/1761] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:14 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:14.609464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.611960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:14.618084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-dd954065-b26b-45a4-b052-757536ea9142 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:14.619676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-dd954065-b26b-45a4-b052-757536ea9142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:14.626275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326ae810-3770-4f00-aa7f-93671af1ec33 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.631730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2886e646-85f9-4ac5-8104-589cedc6b8f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.632485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2886e646-85f9-4ac5-8104-589cedc6b8f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.632832 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2886e646-85f9-4ac5-8104-589cedc6b8f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.633029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2886e646-85f9-4ac5-8104-589cedc6b8f2 admin admin] PATH_INFO: `/v3/projects/b5ae047cf98d43779955e10358da705d/users/4d59ac7f5f6447519d8ec204ab414d5e/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.635171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2886e646-85f9-4ac5-8104-589cedc6b8f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5ae047cf98d43779955e10358da705d, user_id=4d59ac7f5f6447519d8ec204ab414d5e, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.635729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 863/1762] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:14.636713 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd954065-b26b-45a4-b052-757536ea9142 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.638233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2886e646-85f9-4ac5-8104-589cedc6b8f2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.672792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 900/1763] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:14 2022] PUT /identity/v3/projects/b5ae047cf98d43779955e10358da705d/users/4d59ac7f5f6447519d8ec204ab414d5e/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:14.688853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5aa81a-2904-40b8-ae19-84381235ed85 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.689167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5aa81a-2904-40b8-ae19-84381235ed85 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.689429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5aa81a-2904-40b8-ae19-84381235ed85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.690869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd954065-b26b-45a4-b052-757536ea9142 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:14.743691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4a5aa81a-2904-40b8-ae19-84381235ed85 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:14.744964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4a5aa81a-2904-40b8-ae19-84381235ed85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:14.836450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebb28d2-b54f-4fd7-a6ec-2c84c8aa608c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:14.838655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebb28d2-b54f-4fd7-a6ec-2c84c8aa608c admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.839052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebb28d2-b54f-4fd7-a6ec-2c84c8aa608c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.839567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebb28d2-b54f-4fd7-a6ec-2c84c8aa608c admin admin] PATH_INFO: `/v3/projects/ac7c766e14dd43bd9846d6fb3d057b8e/users/3fc253b7db3d4def949f1955dbb60069/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.852419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:14.867399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebb28d2-b54f-4fd7-a6ec-2c84c8aa608c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac7c766e14dd43bd9846d6fb3d057b8e, user_id=3fc253b7db3d4def949f1955dbb60069, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:14.870657 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebb28d2-b54f-4fd7-a6ec-2c84c8aa608c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:14.894019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 864/1764] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:14 2022] PUT /identity/v3/projects/ac7c766e14dd43bd9846d6fb3d057b8e/users/3fc253b7db3d4def949f1955dbb60069/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:14.896842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2212130a-7cc6-4c23-b685-ff4f41a9b03f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:14.896842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2212130a-7cc6-4c23-b685-ff4f41a9b03f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:14.896842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2212130a-7cc6-4c23-b685-ff4f41a9b03f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:14.933860 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 901/1765] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:14.935221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5aa81a-2904-40b8-ae19-84381235ed85 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:14.939959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2212130a-7cc6-4c23-b685-ff4f41a9b03f None None] MFA Rules not processed for user `4d59ac7f5f6447519d8ec204ab414d5e`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:14.944092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2212130a-7cc6-4c23-b685-ff4f41a9b03f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:14.952240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5aa81a-2904-40b8-ae19-84381235ed85 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.078494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 865/1766] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:14 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:15.079794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2212130a-7cc6-4c23-b685-ff4f41a9b03f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.089245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.090180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.090640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.091184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.091846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.094211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.114291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 902/1767] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:14 2022] GET /identity/v3/roles => generated 1420 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:15.116115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.120258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:15.166716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:15.173572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2212130a-7cc6-4c23-b685-ff4f41a9b03f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.178869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.180219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.180543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.180883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.181622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:15.182058 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.185088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.199323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 866/1768] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:15.200433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.201040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9da66c8-a373-4538-841f-3ae5e6ef0e65 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.205984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b13980-d38f-4e79-9e9d-94810787312c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.206592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b13980-d38f-4e79-9e9d-94810787312c service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.206904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b13980-d38f-4e79-9e9d-94810787312c service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.207257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b13980-d38f-4e79-9e9d-94810787312c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.211127 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5566f2e-c3c4-439a-afba-e61a31cc407e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.216231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e343b8-7c84-45b8-ba3c-666635a5b480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.217767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e343b8-7c84-45b8-ba3c-666635a5b480 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.217767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e343b8-7c84-45b8-ba3c-666635a5b480 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.218018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e343b8-7c84-45b8-ba3c-666635a5b480 admin admin] PATH_INFO: `/v3/projects/ac7c766e14dd43bd9846d6fb3d057b8e/users/3fc253b7db3d4def949f1955dbb60069/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.220078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e343b8-7c84-45b8-ba3c-666635a5b480 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac7c766e14dd43bd9846d6fb3d057b8e, user_id=3fc253b7db3d4def949f1955dbb60069, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.222816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e343b8-7c84-45b8-ba3c-666635a5b480 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.229958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b13980-d38f-4e79-9e9d-94810787312c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.231640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b13980-d38f-4e79-9e9d-94810787312c service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.235867 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 867/1769] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:15 2022] PUT /identity/v3/projects/ac7c766e14dd43bd9846d6fb3d057b8e/users/3fc253b7db3d4def949f1955dbb60069/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:15.240265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e343b8-7c84-45b8-ba3c-666635a5b480 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.244844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 903/1770] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:15 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:15.309729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:15.334120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e343b8-7c84-45b8-ba3c-666635a5b480 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.338066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034023ad-6603-4e67-a26d-fb5e783dcb6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.338838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034023ad-6603-4e67-a26d-fb5e783dcb6e admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.339135 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034023ad-6603-4e67-a26d-fb5e783dcb6e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.339400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034023ad-6603-4e67-a26d-fb5e783dcb6e admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.340045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034023ad-6603-4e67-a26d-fb5e783dcb6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.342727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034023ad-6603-4e67-a26d-fb5e783dcb6e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.379669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 868/1771] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:15 2022] POST /identity/v3/projects => generated 371 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:15.383706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b13980-d38f-4e79-9e9d-94810787312c service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.408304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b13980-d38f-4e79-9e9d-94810787312c service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.443700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034023ad-6603-4e67-a26d-fb5e783dcb6e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.558526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034023ad-6603-4e67-a26d-fb5e783dcb6e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.564739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.565038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.565508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.565693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.568761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:15.570385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fa9d8-4b91-43b5-b139-b408c8f740c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.571666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fa9d8-4b91-43b5-b139-b408c8f740c4 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.572296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fa9d8-4b91-43b5-b139-b408c8f740c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.572786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fa9d8-4b91-43b5-b139-b408c8f740c4 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.573764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728fa9d8-4b91-43b5-b139-b408c8f740c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.577350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728fa9d8-4b91-43b5-b139-b408c8f740c4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.614589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 904/1772] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:15 2022] POST /identity/v3/users => generated 286 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:15.616087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fa9d8-4b91-43b5-b139-b408c8f740c4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.633866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fa9d8-4b91-43b5-b139-b408c8f740c4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.642062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d1d9b-874f-44d3-b432-a027d2a7a36f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.642791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d1d9b-874f-44d3-b432-a027d2a7a36f admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.643114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d1d9b-874f-44d3-b432-a027d2a7a36f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.643340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d1d9b-874f-44d3-b432-a027d2a7a36f admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.643986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7d1d9b-874f-44d3-b432-a027d2a7a36f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.645740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7d1d9b-874f-44d3-b432-a027d2a7a36f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.657281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 905/1773] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:15 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:15.661285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d1d9b-874f-44d3-b432-a027d2a7a36f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.664691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.666143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.672627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d1d9b-874f-44d3-b432-a027d2a7a36f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.678733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb93eb8-b180-4de3-a6b2-5e3ca87f8581 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.679393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb93eb8-b180-4de3-a6b2-5e3ca87f8581 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.679677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb93eb8-b180-4de3-a6b2-5e3ca87f8581 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.679957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 869/1774] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:15 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:15.680517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb93eb8-b180-4de3-a6b2-5e3ca87f8581 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.681488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb93eb8-b180-4de3-a6b2-5e3ca87f8581 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.683540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb93eb8-b180-4de3-a6b2-5e3ca87f8581 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.701469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 906/1775] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:15 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:15.702587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.715750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0af195f-3a86-4a13-92bc-14a772e2d53b service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:15.721112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:15.721767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.722046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.722319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] PATH_INFO: `/v3/projects/2db9e6790f864c579bf76ccd5f74010d/users/d857e12439484ca886719051b4929649/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.739835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=2db9e6790f864c579bf76ccd5f74010d, user_id=d857e12439484ca886719051b4929649, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:15.743405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:15.763083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 870/1776] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:15 2022] PUT /identity/v3/projects/2db9e6790f864c579bf76ccd5f74010d/users/d857e12439484ca886719051b4929649/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:15.774323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b4501-c075-41dd-ab81-f324f432d83e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:15.774611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b4501-c075-41dd-ab81-f324f432d83e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:15.774884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b4501-c075-41dd-ab81-f324f432d83e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:15.808993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-856b4501-c075-41dd-ab81-f324f432d83e None None] MFA Rules not processed for user `d857e12439484ca886719051b4929649`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:15.809661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-856b4501-c075-41dd-ab81-f324f432d83e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:15.944608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 907/1777] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:15.954366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:15.956864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:16.037578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e996ec-28d4-4f79-8f18-7daec64ea3cb admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:16.061346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fba08a-1a10-47c8-aa15-3fb2bb002d33 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:16.062222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fba08a-1a10-47c8-aa15-3fb2bb002d33 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:16.062692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fba08a-1a10-47c8-aa15-3fb2bb002d33 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:16.063121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fba08a-1a10-47c8-aa15-3fb2bb002d33 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:16.079245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fba08a-1a10-47c8-aa15-3fb2bb002d33 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:16.082086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fba08a-1a10-47c8-aa15-3fb2bb002d33 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:16.094049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 871/1778] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:15 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:16.122084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856b4501-c075-41dd-ab81-f324f432d83e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:16.165410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:16.224419 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856b4501-c075-41dd-ab81-f324f432d83e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:16.229245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:16.229883 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:16.230157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:16.230479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] PATH_INFO: `/v3/users/d857e12439484ca886719051b4929649` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:16.231708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d857e12439484ca886719051b4929649)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:16.234251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:16.285337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd857e12439484ca886719051b4929649'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:16.292458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd857e12439484ca886719051b4929649'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:16.311122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd857e12439484ca886719051b4929649'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:16.312105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd857e12439484ca886719051b4929649'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:16.312701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-aabd8ea1-9f90-4810-a615-19369ff9dbd5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd857e12439484ca886719051b4929649'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:16.325982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 908/1779] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:16 2022] DELETE /identity/v3/users/d857e12439484ca886719051b4929649 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:16.331798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771e5a13-5943-4cb6-9077-efe1c96725e1 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:16.332460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771e5a13-5943-4cb6-9077-efe1c96725e1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:16.333127 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771e5a13-5943-4cb6-9077-efe1c96725e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:16.363949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-771e5a13-5943-4cb6-9077-efe1c96725e1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:16.365233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-771e5a13-5943-4cb6-9077-efe1c96725e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:16.371467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:16.406575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689a0c4-5e64-48d2-8360-99e39b9cb859 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:16.406940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689a0c4-5e64-48d2-8360-99e39b9cb859 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:16.407113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689a0c4-5e64-48d2-8360-99e39b9cb859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:16.438136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-7689a0c4-5e64-48d2-8360-99e39b9cb859 None None] MFA Rules not processed for user `3fc253b7db3d4def949f1955dbb60069`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:16.439487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7689a0c4-5e64-48d2-8360-99e39b9cb859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:16.440701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 872/1780] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:16.450400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771e5a13-5943-4cb6-9077-efe1c96725e1 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:16.453426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-771e5a13-5943-4cb6-9077-efe1c96725e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:16.538363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771e5a13-5943-4cb6-9077-efe1c96725e1 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:16.542361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:16.542968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:16.543246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:16.543661 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:16.546069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 909/1781] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:16 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:16.553466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7689a0c4-5e64-48d2-8360-99e39b9cb859 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:16.554858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7689a0c4-5e64-48d2-8360-99e39b9cb859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:16.558854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:16.560349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:16.574215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 873/1782] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:16.597253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:16.633202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7689a0c4-5e64-48d2-8360-99e39b9cb859 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:16.637378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496364a4-4422-4704-8bc0-f31302bfd403 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:16.637987 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496364a4-4422-4704-8bc0-f31302bfd403 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:16.638269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496364a4-4422-4704-8bc0-f31302bfd403 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:16.638630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496364a4-4422-4704-8bc0-f31302bfd403 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:16.641251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-496364a4-4422-4704-8bc0-f31302bfd403 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:16.732012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496364a4-4422-4704-8bc0-f31302bfd403 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:16.734165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496364a4-4422-4704-8bc0-f31302bfd403 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:16.752991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 910/1783] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:16 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:16.901626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:16.903323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:16.918215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5812ac24-69cf-4fe7-b702-a63b72f33247 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:16.927109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:16.927747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:16.928202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:16.928353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] PATH_INFO: `/v3/projects/2db9e6790f864c579bf76ccd5f74010d` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:16.930763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2db9e6790f864c579bf76ccd5f74010d)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:16.933545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:17.026520 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2db9e6790f864c579bf76ccd5f74010d', 'request_id': 'req-89eb2a03-c0a8-48ba-90cf-e8cdec972701'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:17.034142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2db9e6790f864c579bf76ccd5f74010d', 'request_id': 'req-89eb2a03-c0a8-48ba-90cf-e8cdec972701'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:17.042814 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2db9e6790f864c579bf76ccd5f74010d', 'request_id': 'req-89eb2a03-c0a8-48ba-90cf-e8cdec972701'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:17.058813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] The token cache is being invalidate because project 2db9e6790f864c579bf76ccd5f74010d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:17.059092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:17.059932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-89eb2a03-c0a8-48ba-90cf-e8cdec972701 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:17.062932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 874/1784] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:16 2022] DELETE /identity/v3/projects/2db9e6790f864c579bf76ccd5f74010d => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:17.068105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48a4ee8-ea77-4948-b9eb-0b307a8aa576 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.068566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48a4ee8-ea77-4948-b9eb-0b307a8aa576 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.068884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48a4ee8-ea77-4948-b9eb-0b307a8aa576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.092475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e48a4ee8-ea77-4948-b9eb-0b307a8aa576 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:17.093957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e48a4ee8-ea77-4948-b9eb-0b307a8aa576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:17.196977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 911/1785] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:17.207356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d829a694-d49b-4873-a83b-2a080756c2ee None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.208596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d829a694-d49b-4873-a83b-2a080756c2ee None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.209181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d829a694-d49b-4873-a83b-2a080756c2ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-d829a694-d49b-4873-a83b-2a080756c2ee None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:17.240820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d829a694-d49b-4873-a83b-2a080756c2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:17.247634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 875/1786] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:17.252532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48a4ee8-ea77-4948-b9eb-0b307a8aa576 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:17.263569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48a4ee8-ea77-4948-b9eb-0b307a8aa576 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:17.268735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:17.269357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.269636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.269914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.270573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:17.270881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:17.272940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:17.289469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 912/1787] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:52:17.291011 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d829a694-d49b-4873-a83b-2a080756c2ee None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:17.311037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d829a694-d49b-4873-a83b-2a080756c2ee None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:17.318276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bb45d1-b203-4aff-9256-854747b321e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:17.319297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb45d1-b203-4aff-9256-854747b321e9 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.319831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb45d1-b203-4aff-9256-854747b321e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.320417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb45d1-b203-4aff-9256-854747b321e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.321483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bb45d1-b203-4aff-9256-854747b321e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:17.325273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bb45d1-b203-4aff-9256-854747b321e9 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:17.368477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 876/1788] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:17 2022] POST /identity/v3/projects => generated 373 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:17.371486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:17.384237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8fc19c-4f85-480c-ba3a-fa6bcc89ac14 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:17.488744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7704df74-0a32-411b-a1b0-1bbf36a733e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:17.489327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7704df74-0a32-411b-a1b0-1bbf36a733e8 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.489613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7704df74-0a32-411b-a1b0-1bbf36a733e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.489885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7704df74-0a32-411b-a1b0-1bbf36a733e8 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.490796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7704df74-0a32-411b-a1b0-1bbf36a733e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:17.492933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7704df74-0a32-411b-a1b0-1bbf36a733e8 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:17.532073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 913/1789] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:17 2022] POST /identity/v3/users => generated 287 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:17.538342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bb45d1-b203-4aff-9256-854747b321e9 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:17.551957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bb45d1-b203-4aff-9256-854747b321e9 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:17.559645 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06edcd-2049-4900-86bf-64b1e78c2018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:17.560773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06edcd-2049-4900-86bf-64b1e78c2018 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.561205 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06edcd-2049-4900-86bf-64b1e78c2018 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.561571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06edcd-2049-4900-86bf-64b1e78c2018 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.562437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b06edcd-2049-4900-86bf-64b1e78c2018 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:17.564779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b06edcd-2049-4900-86bf-64b1e78c2018 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:17.567856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:17.586613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 877/1790] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:17 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:17.591520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7704df74-0a32-411b-a1b0-1bbf36a733e8 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:17.600116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:17.604743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7704df74-0a32-411b-a1b0-1bbf36a733e8 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:17.612372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97950177-1125-46bd-82e4-8e7cd30a0bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:17.613533 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97950177-1125-46bd-82e4-8e7cd30a0bef admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.613824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97950177-1125-46bd-82e4-8e7cd30a0bef admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.614232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97950177-1125-46bd-82e4-8e7cd30a0bef admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.614909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97950177-1125-46bd-82e4-8e7cd30a0bef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:17.617751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97950177-1125-46bd-82e4-8e7cd30a0bef admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:17.634100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 914/1791] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:17 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:17.638517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06edcd-2049-4900-86bf-64b1e78c2018 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:17.650824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06edcd-2049-4900-86bf-64b1e78c2018 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:17.658185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:17.659463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.659463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.659463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] PATH_INFO: `/v3/projects/72ee7f45fb4d407eaa298629ab8a8526/users/84e3b3883e3b4f2b8e2bf52d87fbc77f/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.677362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72ee7f45fb4d407eaa298629ab8a8526, user_id=84e3b3883e3b4f2b8e2bf52d87fbc77f, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:17.680023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:17.702501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 878/1792] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:17 2022] PUT /identity/v3/projects/72ee7f45fb4d407eaa298629ab8a8526/users/84e3b3883e3b4f2b8e2bf52d87fbc77f/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:17.713024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43072380-5751-49ad-966b-a96689019e82 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.713242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43072380-5751-49ad-966b-a96689019e82 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.713516 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43072380-5751-49ad-966b-a96689019e82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.744732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-43072380-5751-49ad-966b-a96689019e82 None None] MFA Rules not processed for user `84e3b3883e3b4f2b8e2bf52d87fbc77f`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:17.745970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-43072380-5751-49ad-966b-a96689019e82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:17.820735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 915/1793] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:17.834166 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:17.836862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:17.910019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16475835-0d15-4d40-8c61-3cca553fec75 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:17.914713 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:17.915340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:17.915615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:17.915885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:17.926869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:17.928360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:17.939153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 879/1794] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:18.243004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43072380-5751-49ad-966b-a96689019e82 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:18.254154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43072380-5751-49ad-966b-a96689019e82 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:18.258801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd33457b-fbfe-4d7b-bb01-67650f7fe5e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:18.259558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd33457b-fbfe-4d7b-bb01-67650f7fe5e8 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:18.259908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd33457b-fbfe-4d7b-bb01-67650f7fe5e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:18.260253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd33457b-fbfe-4d7b-bb01-67650f7fe5e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:18.262217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bd33457b-fbfe-4d7b-bb01-67650f7fe5e8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:18.343420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd33457b-fbfe-4d7b-bb01-67650f7fe5e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:18.344993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd33457b-fbfe-4d7b-bb01-67650f7fe5e8 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:18.357268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 916/1795] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:18 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:20.258302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:20.260425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:20.343151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bce67b-6238-4424-8cc2-2c5d93211c3a service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:20.348090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:20.348768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:20.349043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:20.349305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] PATH_INFO: `/v3/users/ba9eb845217f46159e9a41e157f7e62b` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:20.350495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba9eb845217f46159e9a41e157f7e62b)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:20.352465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:20.415501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba9eb845217f46159e9a41e157f7e62b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:20.423433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba9eb845217f46159e9a41e157f7e62b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:20.435963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba9eb845217f46159e9a41e157f7e62b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:20.436752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba9eb845217f46159e9a41e157f7e62b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:20.437412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba9eb845217f46159e9a41e157f7e62b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:20.441552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:20.459687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 880/1796] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:20 2022] DELETE /identity/v3/users/ba9eb845217f46159e9a41e157f7e62b => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:20.465256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc47b491-a1de-44d6-a117-29751841838b None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:20.465733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc47b491-a1de-44d6-a117-29751841838b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:20.466150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc47b491-a1de-44d6-a117-29751841838b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:20.488200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-bc47b491-a1de-44d6-a117-29751841838b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:20.489422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bc47b491-a1de-44d6-a117-29751841838b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:20.552852 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 917/1797] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:20.562707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:20.564185 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:20.624938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359586a9-3602-404e-953b-f0781134faf0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:20.628898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:20.629514 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:20.629800 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:20.630078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:20.641721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:20.643356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:20.655344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 881/1798] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:20.918676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc47b491-a1de-44d6-a117-29751841838b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:20.920295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bc47b491-a1de-44d6-a117-29751841838b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:20.933315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc47b491-a1de-44d6-a117-29751841838b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:20.938959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:20.939608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:20.939881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:20.940187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] PATH_INFO: `/v3/users/59da7a35ed8043c49a1482dc94a1e83b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:20.941375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] RBAC: Authorizing `identity:delete_user(user_id=59da7a35ed8043c49a1482dc94a1e83b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:20.943353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:20.960450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:21.019498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:21.022102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:21.022511 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59da7a35ed8043c49a1482dc94a1e83b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.040022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59da7a35ed8043c49a1482dc94a1e83b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.062458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59da7a35ed8043c49a1482dc94a1e83b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.063370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59da7a35ed8043c49a1482dc94a1e83b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.063527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5715350c-3e68-4871-82c9-56c1c21d4ca1 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:21.063988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-39270d4b-4a32-47fb-a9ee-5ea9e7e8659c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59da7a35ed8043c49a1482dc94a1e83b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.071409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:21.072063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.072584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.072812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] PATH_INFO: `/v3/projects/257c15df232146cab06af2a01271cf47` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.073973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] RBAC: Authorizing `identity:delete_project(project_id=257c15df232146cab06af2a01271cf47)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:21.078095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:21.093101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 918/1799] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:20 2022] DELETE /identity/v3/users/59da7a35ed8043c49a1482dc94a1e83b => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:21.098167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7538a568-b4ff-404f-9662-4aeb02c47bcc None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.098840 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7538a568-b4ff-404f-9662-4aeb02c47bcc None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.099281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7538a568-b4ff-404f-9662-4aeb02c47bcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.125139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-7538a568-b4ff-404f-9662-4aeb02c47bcc None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:21.128750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7538a568-b4ff-404f-9662-4aeb02c47bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:21.153613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '257c15df232146cab06af2a01271cf47', 'request_id': 'req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.162294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '257c15df232146cab06af2a01271cf47', 'request_id': 'req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.169374 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '257c15df232146cab06af2a01271cf47', 'request_id': 'req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.195324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] The token cache is being invalidate because project 257c15df232146cab06af2a01271cf47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:21.196058 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.197222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e7ee2820-63f2-462d-9f0d-ef0eaa572a0d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.204233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 882/1800] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:21 2022] DELETE /identity/v3/projects/257c15df232146cab06af2a01271cf47 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:21.208010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbee7e-6725-4e16-81fa-8bd045bb3a93 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.208479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbee7e-6725-4e16-81fa-8bd045bb3a93 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.208993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbee7e-6725-4e16-81fa-8bd045bb3a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.259107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 919/1801] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:21.259209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-99bbee7e-6725-4e16-81fa-8bd045bb3a93 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:21.259209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-99bbee7e-6725-4e16-81fa-8bd045bb3a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:21.261636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7538a568-b4ff-404f-9662-4aeb02c47bcc None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:21.268698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7538a568-b4ff-404f-9662-4aeb02c47bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:21.295689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 883/1802] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:21.296760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bbee7e-6725-4e16-81fa-8bd045bb3a93 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:21.307250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-99bbee7e-6725-4e16-81fa-8bd045bb3a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:21.335443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bbee7e-6725-4e16-81fa-8bd045bb3a93 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:21.343880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:21.344567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.345010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.346295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] PATH_INFO: `/v3/users/84e3b3883e3b4f2b8e2bf52d87fbc77f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.348201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] RBAC: Authorizing `identity:delete_user(user_id=84e3b3883e3b4f2b8e2bf52d87fbc77f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:21.350182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:21.444716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84e3b3883e3b4f2b8e2bf52d87fbc77f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.454420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84e3b3883e3b4f2b8e2bf52d87fbc77f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.456711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7538a568-b4ff-404f-9662-4aeb02c47bcc None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:21.466359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f1ab44-4f68-4e3e-9091-ad2283f59f55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:21.468510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1ab44-4f68-4e3e-9091-ad2283f59f55 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.468797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1ab44-4f68-4e3e-9091-ad2283f59f55 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.470830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1ab44-4f68-4e3e-9091-ad2283f59f55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.474106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:21.486885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f1ab44-4f68-4e3e-9091-ad2283f59f55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:21.488720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f1ab44-4f68-4e3e-9091-ad2283f59f55 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:21.500425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84e3b3883e3b4f2b8e2bf52d87fbc77f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.501396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84e3b3883e3b4f2b8e2bf52d87fbc77f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.502103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6e537d20-d7ad-4e85-8ed2-313720bd453e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84e3b3883e3b4f2b8e2bf52d87fbc77f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:21.524043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 920/1803] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:21.528263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 884/1804] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:21 2022] DELETE /identity/v3/users/84e3b3883e3b4f2b8e2bf52d87fbc77f => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:21.532748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121037a-735a-48ad-a803-1261ec698aa9 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.533153 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121037a-735a-48ad-a803-1261ec698aa9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.533606 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993bb7be-3004-4f45-9508-d447e1806726 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.534070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993bb7be-3004-4f45-9508-d447e1806726 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.534184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121037a-735a-48ad-a803-1261ec698aa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.534576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993bb7be-3004-4f45-9508-d447e1806726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.570674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f121037a-735a-48ad-a803-1261ec698aa9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:21.571337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-993bb7be-3004-4f45-9508-d447e1806726 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:21.572292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f121037a-735a-48ad-a803-1261ec698aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:21.575264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-993bb7be-3004-4f45-9508-d447e1806726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:21.735494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:21.776637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 885/1805] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:21.790492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993bb7be-3004-4f45-9508-d447e1806726 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:21.792640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-993bb7be-3004-4f45-9508-d447e1806726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:21.836620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 921/1806] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:21.844162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121037a-735a-48ad-a803-1261ec698aa9 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:21.860069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121037a-735a-48ad-a803-1261ec698aa9 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:21.868362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:21.869382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.869828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.870233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.871308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:21.871696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:21.875391 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:21.898470 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993bb7be-3004-4f45-9508-d447e1806726 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:21.905140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e77770-996c-4792-a0ce-99eff2ec63d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:21.905885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e77770-996c-4792-a0ce-99eff2ec63d7 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.906166 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e77770-996c-4792-a0ce-99eff2ec63d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.906421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e77770-996c-4792-a0ce-99eff2ec63d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.914239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 922/1807] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:21.916230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:21.919585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e77770-996c-4792-a0ce-99eff2ec63d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:21.922181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e77770-996c-4792-a0ce-99eff2ec63d7 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:21.927331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb4f8c-98c6-4f51-9217-357e5e95d5c2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:21.932761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6571bd1-336e-4179-b26d-e33e9423305f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:21.933393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6571bd1-336e-4179-b26d-e33e9423305f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:21.933765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6571bd1-336e-4179-b26d-e33e9423305f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:21.934049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6571bd1-336e-4179-b26d-e33e9423305f admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:21.934718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6571bd1-336e-4179-b26d-e33e9423305f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:21.937690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6571bd1-336e-4179-b26d-e33e9423305f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:21.942945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 886/1808] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:21.994479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 923/1809] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:21 2022] POST /identity/v3/projects => generated 365 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:22.001030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e77770-996c-4792-a0ce-99eff2ec63d7 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:22.022415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e77770-996c-4792-a0ce-99eff2ec63d7 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:22.116269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f034f2-5a03-41e5-9370-9e2668a4a1de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:22.116984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f034f2-5a03-41e5-9370-9e2668a4a1de admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:22.117327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f034f2-5a03-41e5-9370-9e2668a4a1de admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:22.117664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f034f2-5a03-41e5-9370-9e2668a4a1de admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:22.118364 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f034f2-5a03-41e5-9370-9e2668a4a1de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:22.120762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f034f2-5a03-41e5-9370-9e2668a4a1de admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:22.153259 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6571bd1-336e-4179-b26d-e33e9423305f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:22.161089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 887/1810] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:21 2022] POST /identity/v3/users => generated 283 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:22.165811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6571bd1-336e-4179-b26d-e33e9423305f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:22.178685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025a27be-6851-48ec-b054-67585c1e0bc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:22.180688 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025a27be-6851-48ec-b054-67585c1e0bc2 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:22.183717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025a27be-6851-48ec-b054-67585c1e0bc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:22.183717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025a27be-6851-48ec-b054-67585c1e0bc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:22.184436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025a27be-6851-48ec-b054-67585c1e0bc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:22.186216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025a27be-6851-48ec-b054-67585c1e0bc2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:22.206535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 924/1811] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:22 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:22.210756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f034f2-5a03-41e5-9370-9e2668a4a1de admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:22.234781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f034f2-5a03-41e5-9370-9e2668a4a1de admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:22.246829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f6394b-33d7-44ba-bb6d-c7c4e2b7ca99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:22.247426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f6394b-33d7-44ba-bb6d-c7c4e2b7ca99 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:22.248547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f6394b-33d7-44ba-bb6d-c7c4e2b7ca99 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:22.248547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f6394b-33d7-44ba-bb6d-c7c4e2b7ca99 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:22.249224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f6394b-33d7-44ba-bb6d-c7c4e2b7ca99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:22.251515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f6394b-33d7-44ba-bb6d-c7c4e2b7ca99 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:22.276214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 888/1812] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:22 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:22.281874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025a27be-6851-48ec-b054-67585c1e0bc2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:22.294456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025a27be-6851-48ec-b054-67585c1e0bc2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:22.301575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:22.302288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:22.302546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:22.303150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] PATH_INFO: `/v3/projects/c6f967be521f4de6afbddd1eae7ad283/users/781bb86257534a7980b0ba1f29996b22/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:22.323980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6f967be521f4de6afbddd1eae7ad283, user_id=781bb86257534a7980b0ba1f29996b22, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:22.328210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:22.356772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 925/1813] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:22 2022] PUT /identity/v3/projects/c6f967be521f4de6afbddd1eae7ad283/users/781bb86257534a7980b0ba1f29996b22/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:22.361137 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576af56-d77c-4356-8776-6865e2dca2c5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:22.361137 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576af56-d77c-4356-8776-6865e2dca2c5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:22.363163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576af56-d77c-4356-8776-6865e2dca2c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:22.402778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:22.428670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f576af56-d77c-4356-8776-6865e2dca2c5 None None] MFA Rules not processed for user `781bb86257534a7980b0ba1f29996b22`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:22.431960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f576af56-d77c-4356-8776-6865e2dca2c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:22.557382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 889/1814] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:22 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:22.577829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:22.585001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f576af56-d77c-4356-8776-6865e2dca2c5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:22.586819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:22.589278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f576af56-d77c-4356-8776-6865e2dca2c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:22.761557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01927d1-5989-4f8b-91b6-0519d15a1343 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:22.771095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:22.773040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:22.774011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:22.774011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:22.783486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:22.785103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:22.788056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f576af56-d77c-4356-8776-6865e2dca2c5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:22.808253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 926/1815] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:22 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:22.809494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:22.811992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:22.817085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:22.817888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:22.818171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:22.818446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] PATH_INFO: `/v3/projects/72ee7f45fb4d407eaa298629ab8a8526` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:22.824019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72ee7f45fb4d407eaa298629ab8a8526)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:22.829961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:22.853275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b1e0c1-4fd4-4e6c-a21d-5356fa82987d service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:22.865633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:22.869306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:22.874103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:22.874564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] PATH_INFO: `/v3/projects/abc092e5568b4a3cb5179c274198fd60` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:22.880615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] RBAC: Authorizing `identity:delete_project(project_id=abc092e5568b4a3cb5179c274198fd60)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:22.882542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:22.953375 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72ee7f45fb4d407eaa298629ab8a8526', 'request_id': 'req-a0d97b27-f8f7-479c-badd-75495a5d3d43'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:22.965035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72ee7f45fb4d407eaa298629ab8a8526', 'request_id': 'req-a0d97b27-f8f7-479c-badd-75495a5d3d43'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:22.991435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72ee7f45fb4d407eaa298629ab8a8526', 'request_id': 'req-a0d97b27-f8f7-479c-badd-75495a5d3d43'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.018947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abc092e5568b4a3cb5179c274198fd60', 'request_id': 'req-8f03c77e-47c5-4d6b-a842-d458ce217158'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.024068 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] The token cache is being invalidate because project 72ee7f45fb4d407eaa298629ab8a8526 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:23.024394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.026041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.028845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abc092e5568b4a3cb5179c274198fd60', 'request_id': 'req-8f03c77e-47c5-4d6b-a842-d458ce217158'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.032404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 890/1816] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:22 2022] DELETE /identity/v3/projects/72ee7f45fb4d407eaa298629ab8a8526 => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:23.071851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abc092e5568b4a3cb5179c274198fd60', 'request_id': 'req-8f03c77e-47c5-4d6b-a842-d458ce217158'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.073726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:23.096939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] The token cache is being invalidate because project abc092e5568b4a3cb5179c274198fd60 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:23.097311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.098116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.100794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 927/1817] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:22 2022] DELETE /identity/v3/projects/abc092e5568b4a3cb5179c274198fd60 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:23.109643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:23.111669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:23.271321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:23.398547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d97b27-f8f7-479c-badd-75495a5d3d43 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:23.511551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:23.512415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:23.512730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:23.513056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] PATH_INFO: `/v3/users/4d59ac7f5f6447519d8ec204ab414d5e` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:23.517568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d59ac7f5f6447519d8ec204ab414d5e)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:23.520720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:23.628273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d59ac7f5f6447519d8ec204ab414d5e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.637278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d59ac7f5f6447519d8ec204ab414d5e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.657306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d59ac7f5f6447519d8ec204ab414d5e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.658248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d59ac7f5f6447519d8ec204ab414d5e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.659507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f08c3601-8705-4e52-a59e-efc0f3af0f09 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d59ac7f5f6447519d8ec204ab414d5e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:23.675729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 891/1818] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:23 2022] DELETE /identity/v3/users/4d59ac7f5f6447519d8ec204ab414d5e => generated 0 bytes in 571 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:23.993101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:23.994866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:24.015461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f03c77e-47c5-4d6b-a842-d458ce217158 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:24.098346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:24.098983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.099256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.099526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] PATH_INFO: `/v3/projects/b5ae047cf98d43779955e10358da705d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.100760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5ae047cf98d43779955e10358da705d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:24.102640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:24.160455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5ae047cf98d43779955e10358da705d', 'request_id': 'req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:24.167582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5ae047cf98d43779955e10358da705d', 'request_id': 'req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:24.177629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5ae047cf98d43779955e10358da705d', 'request_id': 'req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:24.189425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] The token cache is being invalidate because project b5ae047cf98d43779955e10358da705d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:24.189720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:24.190392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2bc1fc3a-0a44-4496-b3b8-d846921ee60b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:24.192090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 928/1819] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:23 2022] DELETE /identity/v3/projects/b5ae047cf98d43779955e10358da705d => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:24.199260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa43e8-caeb-46f7-909d-d191153cdb92 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.199720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa43e8-caeb-46f7-909d-d191153cdb92 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.200240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa43e8-caeb-46f7-909d-d191153cdb92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.230365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3baa43e8-caeb-46f7-909d-d191153cdb92 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:24.233131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3baa43e8-caeb-46f7-909d-d191153cdb92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:24.305446 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:24.322527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 892/1820] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:24.334378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc411623-bc85-4aa1-9044-6702eec75692 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.334845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc411623-bc85-4aa1-9044-6702eec75692 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.335298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc411623-bc85-4aa1-9044-6702eec75692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.365032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-dc411623-bc85-4aa1-9044-6702eec75692 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:24.366726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-dc411623-bc85-4aa1-9044-6702eec75692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:24.377980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 929/1821] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:24.382887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa43e8-caeb-46f7-909d-d191153cdb92 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:24.394360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa43e8-caeb-46f7-909d-d191153cdb92 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:24.399861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:24.399861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.400581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.400581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.401129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:24.401434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:24.403608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:24.418265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 893/1822] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:24.423464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc411623-bc85-4aa1-9044-6702eec75692 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:24.433165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc411623-bc85-4aa1-9044-6702eec75692 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:24.438437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661e607f-d894-444e-bb8d-2b67c728acb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:24.439048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e607f-d894-444e-bb8d-2b67c728acb2 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.439339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e607f-d894-444e-bb8d-2b67c728acb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.439590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e607f-d894-444e-bb8d-2b67c728acb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.440245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661e607f-d894-444e-bb8d-2b67c728acb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:24.442148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661e607f-d894-444e-bb8d-2b67c728acb2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:24.480174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 930/1823] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:24 2022] POST /identity/v3/projects => generated 367 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:24.484264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:24.500159 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15232d45-8593-4e72-a071-1de9a157a3ba admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:24.597770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5966742-97fd-477a-b0c9-0de6e4a0246f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:24.598532 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5966742-97fd-477a-b0c9-0de6e4a0246f admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.598916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5966742-97fd-477a-b0c9-0de6e4a0246f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.599295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5966742-97fd-477a-b0c9-0de6e4a0246f admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.600157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5966742-97fd-477a-b0c9-0de6e4a0246f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:24.602839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5966742-97fd-477a-b0c9-0de6e4a0246f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:24.631848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 894/1824] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:24 2022] POST /identity/v3/users => generated 284 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:24.636048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661e607f-d894-444e-bb8d-2b67c728acb2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:24.655509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661e607f-d894-444e-bb8d-2b67c728acb2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:24.663584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431d090-832f-455e-905b-e34b69cd3e7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:24.664309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431d090-832f-455e-905b-e34b69cd3e7b admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.664566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431d090-832f-455e-905b-e34b69cd3e7b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.664835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431d090-832f-455e-905b-e34b69cd3e7b admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.665478 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3431d090-832f-455e-905b-e34b69cd3e7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:24.667449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3431d090-832f-455e-905b-e34b69cd3e7b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:24.684513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 931/1825] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:24 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:24.689494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5966742-97fd-477a-b0c9-0de6e4a0246f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:24.700284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5966742-97fd-477a-b0c9-0de6e4a0246f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:24.705370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e03855-d094-40e9-9a7c-e24aa54ce5f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:24.705993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e03855-d094-40e9-9a7c-e24aa54ce5f3 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.706270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e03855-d094-40e9-9a7c-e24aa54ce5f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.706687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e03855-d094-40e9-9a7c-e24aa54ce5f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.707388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e03855-d094-40e9-9a7c-e24aa54ce5f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:24.709216 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e03855-d094-40e9-9a7c-e24aa54ce5f3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:24.726871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 895/1826] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:24 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:24.732121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431d090-832f-455e-905b-e34b69cd3e7b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:24.745376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431d090-832f-455e-905b-e34b69cd3e7b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:24.761371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:24.762100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.762454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.762816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] PATH_INFO: `/v3/projects/b676c58e09dc431aafe91d0535cdf2cc/users/da5be68da2d84ce0b2434d667869fa8c/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.795507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b676c58e09dc431aafe91d0535cdf2cc, user_id=da5be68da2d84ce0b2434d667869fa8c, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:24.801243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:24.820077 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 932/1827] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:24 2022] PUT /identity/v3/projects/b676c58e09dc431aafe91d0535cdf2cc/users/da5be68da2d84ce0b2434d667869fa8c/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:24.843054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c277cfd8-1026-4849-89e8-f97b374cbad3 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:24.843663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c277cfd8-1026-4849-89e8-f97b374cbad3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:24.844010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c277cfd8-1026-4849-89e8-f97b374cbad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:24.881683 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c277cfd8-1026-4849-89e8-f97b374cbad3 None None] MFA Rules not processed for user `da5be68da2d84ce0b2434d667869fa8c`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:24.882939 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c277cfd8-1026-4849-89e8-f97b374cbad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:24.965354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 896/1828] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:24 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:24.984620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:24.987750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:25.012119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:DELETE=9,keystone:SELECT=88,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:25.063243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6eba404-76d6-41f0-b6fd-a769230f972c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:25.068713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7a8a3d-2cdd-400b-a018-f6e3cfe4a56a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a4fd11d5d85479c9aa1d87b23a13141', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:25.068713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a8a3d-2cdd-400b-a018-f6e3cfe4a56a service glance] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:25.068713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a8a3d-2cdd-400b-a018-f6e3cfe4a56a service glance] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:25.069185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a8a3d-2cdd-400b-a018-f6e3cfe4a56a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:25.081004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7a8a3d-2cdd-400b-a018-f6e3cfe4a56a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:25.082427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7a8a3d-2cdd-400b-a018-f6e3cfe4a56a service glance] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:25.093562 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 933/1829] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:24 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:33.401320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a7fd91-88b9-46c3-9268-cde521801fb8 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:33.403129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a7fd91-88b9-46c3-9268-cde521801fb8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:33.403915 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a7fd91-88b9-46c3-9268-cde521801fb8 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:33.407833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 897/1830] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:52:33 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:52:33.438374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e470842-3e5b-4761-9d17-79428510ba49 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:33.438918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e470842-3e5b-4761-9d17-79428510ba49 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:33.439402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e470842-3e5b-4761-9d17-79428510ba49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:33.486626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-9e470842-3e5b-4761-9d17-79428510ba49 None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:33.488610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9e470842-3e5b-4761-9d17-79428510ba49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:33.623797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 934/1831] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:52:33 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:33.640706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c9164-5cd3-454b-8810-bb6d47532357 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:33.641189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c9164-5cd3-454b-8810-bb6d47532357 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:33.641673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c9164-5cd3-454b-8810-bb6d47532357 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:33.645474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 898/1832] 10.222.0.39 () {54 vars in 961 bytes} [Mon Oct 24 08:52:33 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:52:33.652263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b0091f-7632-49a0-8f55-6fa67493017e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:33.652995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b0091f-7632-49a0-8f55-6fa67493017e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:33.653466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b0091f-7632-49a0-8f55-6fa67493017e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:33.707700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d1b0091f-7632-49a0-8f55-6fa67493017e None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:33.716083 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d1b0091f-7632-49a0-8f55-6fa67493017e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:33.728638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 935/1833] 10.222.0.39 () {58 vars in 1058 bytes} [Mon Oct 24 08:52:33 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:33.737824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34c9164-5cd3-454b-8810-bb6d47532357 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:33.755951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34c9164-5cd3-454b-8810-bb6d47532357 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:33.764196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd7d4c8-1940-4003-9861-6b392cdfad55 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:33.765046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd7d4c8-1940-4003-9861-6b392cdfad55 service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:33.765509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd7d4c8-1940-4003-9861-6b392cdfad55 service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:33.766114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd7d4c8-1940-4003-9861-6b392cdfad55 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:33.776922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd7d4c8-1940-4003-9861-6b392cdfad55 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:33.779888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd7d4c8-1940-4003-9861-6b392cdfad55 service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:33.793115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 899/1834] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:34.651871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8067c5e-b9cb-4a21-a762-d9fe12d1bebd None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:34.652772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8067c5e-b9cb-4a21-a762-d9fe12d1bebd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:34.652772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8067c5e-b9cb-4a21-a762-d9fe12d1bebd None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:34.655165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 936/1835] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:52:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:52:34.662603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd911bc-82ad-429f-9ea3-57f4e47d0bb6 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:34.663152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd911bc-82ad-429f-9ea3-57f4e47d0bb6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:34.663152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd911bc-82ad-429f-9ea3-57f4e47d0bb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:34.685440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9dd911bc-82ad-429f-9ea3-57f4e47d0bb6 None None] MFA Rules not processed for user `0c28fe744b614210a21aca0a7395089f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:34.686820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9dd911bc-82ad-429f-9ea3-57f4e47d0bb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:34.695965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 900/1836] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:52:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:34.724330 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8067c5e-b9cb-4a21-a762-d9fe12d1bebd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:34.739959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8067c5e-b9cb-4a21-a762-d9fe12d1bebd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:34.751251 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9be10c-8750-4302-a92a-8499ba96be7f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:34.751885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9be10c-8750-4302-a92a-8499ba96be7f service placement] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:34.752202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9be10c-8750-4302-a92a-8499ba96be7f service placement] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:34.752488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9be10c-8750-4302-a92a-8499ba96be7f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:34.772390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9be10c-8750-4302-a92a-8499ba96be7f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:34.774092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9be10c-8750-4302-a92a-8499ba96be7f service placement] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:34.785681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 937/1837] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:34.829539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd911bc-82ad-429f-9ea3-57f4e47d0bb6 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:34.831035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9dd911bc-82ad-429f-9ea3-57f4e47d0bb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:34.898257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:34.948044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd911bc-82ad-429f-9ea3-57f4e47d0bb6 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:34.954263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:34.955009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:34.955294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:34.955569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] PATH_INFO: `/v3/users/32d8eb0025ec43ce8d967ece7d2f0dd3` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:34.956849 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=32d8eb0025ec43ce8d967ece7d2f0dd3)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:34.958833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:35.026808 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32d8eb0025ec43ce8d967ece7d2f0dd3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.037806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32d8eb0025ec43ce8d967ece7d2f0dd3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.061730 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32d8eb0025ec43ce8d967ece7d2f0dd3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.062716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32d8eb0025ec43ce8d967ece7d2f0dd3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.063425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32d8eb0025ec43ce8d967ece7d2f0dd3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.092843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 901/1838] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:34 2022] DELETE /identity/v3/users/32d8eb0025ec43ce8d967ece7d2f0dd3 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:35.374055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9be10c-8750-4302-a92a-8499ba96be7f service placement] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:35.391776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9be10c-8750-4302-a92a-8499ba96be7f service placement] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:35.527337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:35.528013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:35.528736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:35.529038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] PATH_INFO: `/v3/projects/69f8ba90c84748bda85edc2c7dcd4f73` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:35.530537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69f8ba90c84748bda85edc2c7dcd4f73)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:35.532814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:35.603468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:35.618353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69f8ba90c84748bda85edc2c7dcd4f73', 'request_id': 'req-112e805d-4f6c-465c-9f31-26da24ad8391'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.627473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69f8ba90c84748bda85edc2c7dcd4f73', 'request_id': 'req-112e805d-4f6c-465c-9f31-26da24ad8391'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.636710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69f8ba90c84748bda85edc2c7dcd4f73', 'request_id': 'req-112e805d-4f6c-465c-9f31-26da24ad8391'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.657181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] The token cache is being invalidate because project 69f8ba90c84748bda85edc2c7dcd4f73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:35.658205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.658355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.660200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 938/1839] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:35 2022] DELETE /identity/v3/projects/69f8ba90c84748bda85edc2c7dcd4f73 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:35.665291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:35.667670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:35.683234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c2aa9a-e5b4-40ed-86a9-600452737d6f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:35.768301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:35.769156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:35.769536 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:35.769909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] PATH_INFO: `/v3/users/3fc253b7db3d4def949f1955dbb60069` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:35.771259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3fc253b7db3d4def949f1955dbb60069)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:35.773348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:35.839379 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3fc253b7db3d4def949f1955dbb60069'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.847323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:35.851553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3fc253b7db3d4def949f1955dbb60069'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.878490 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3fc253b7db3d4def949f1955dbb60069'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.879409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3fc253b7db3d4def949f1955dbb60069'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.880168 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-142a15ab-f690-429a-bbfb-5bd6cdbf223e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3fc253b7db3d4def949f1955dbb60069'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:35.897919 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 902/1840] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:35 2022] DELETE /identity/v3/users/3fc253b7db3d4def949f1955dbb60069 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:36.267888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:36.269810 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:36.300729 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e805d-4f6c-465c-9f31-26da24ad8391 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:36.392909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:36.393488 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:36.393791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:36.394022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] PATH_INFO: `/v3/projects/ac7c766e14dd43bd9846d6fb3d057b8e` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:36.395227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac7c766e14dd43bd9846d6fb3d057b8e)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:36.397264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:36.479434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac7c766e14dd43bd9846d6fb3d057b8e', 'request_id': 'req-c05e1c9c-7725-4db5-acd2-f5c812a2992b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.490124 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac7c766e14dd43bd9846d6fb3d057b8e', 'request_id': 'req-c05e1c9c-7725-4db5-acd2-f5c812a2992b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.500342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac7c766e14dd43bd9846d6fb3d057b8e', 'request_id': 'req-c05e1c9c-7725-4db5-acd2-f5c812a2992b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.521058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] The token cache is being invalidate because project ac7c766e14dd43bd9846d6fb3d057b8e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:36.521058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.534315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c05e1c9c-7725-4db5-acd2-f5c812a2992b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.541211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 939/1841] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:36 2022] DELETE /identity/v3/projects/ac7c766e14dd43bd9846d6fb3d057b8e => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:36.548600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789aec89-318d-410f-bf5a-581d8813ac90 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:36.548898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789aec89-318d-410f-bf5a-581d8813ac90 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:36.549176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789aec89-318d-410f-bf5a-581d8813ac90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:36.588832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-789aec89-318d-410f-bf5a-581d8813ac90 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:36.592585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-789aec89-318d-410f-bf5a-581d8813ac90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:36.688797 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 903/1842] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:36.697271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e7db2-8a62-4411-97e1-7952bb061c82 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:36.697837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e7db2-8a62-4411-97e1-7952bb061c82 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:36.697837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e7db2-8a62-4411-97e1-7952bb061c82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:36.727457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789aec89-318d-410f-bf5a-581d8813ac90 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:36.729353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ae8e7db2-8a62-4411-97e1-7952bb061c82 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:36.730136 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-789aec89-318d-410f-bf5a-581d8813ac90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:36.730850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ae8e7db2-8a62-4411-97e1-7952bb061c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:36.741335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 940/1843] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:36.752681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e7db2-8a62-4411-97e1-7952bb061c82 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:36.753372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789aec89-318d-410f-bf5a-581d8813ac90 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:36.758572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:36.759481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:36.759760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:36.760041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] PATH_INFO: `/v3/users/701e4507214f4634ad09e32e21749a35` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:36.761248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=701e4507214f4634ad09e32e21749a35)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:36.763315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:36.764329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e7db2-8a62-4411-97e1-7952bb061c82 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:36.769462 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:36.770091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:36.770373 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:36.770647 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:36.771321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:36.772066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:36.773803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:36.789221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 941/1844] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:36.792039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:36.803183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95069d0-c96e-4276-b9c8-cfa6c34b79a2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:36.808530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:36.809138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:36.809430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:36.809741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:36.811449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:36.813156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:36.834062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '701e4507214f4634ad09e32e21749a35'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.842637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '701e4507214f4634ad09e32e21749a35'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.856022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '701e4507214f4634ad09e32e21749a35'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.857414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '701e4507214f4634ad09e32e21749a35'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.858143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '701e4507214f4634ad09e32e21749a35'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:36.858922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 942/1845] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:36 2022] POST /identity/v3/projects => generated 383 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:36.859893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:36.861680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:36.875047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 904/1846] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:36 2022] DELETE /identity/v3/users/701e4507214f4634ad09e32e21749a35 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:36.881384 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:36.971215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e95df1-e45c-42cb-8631-bd0498739064 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.072839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.073467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.073748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.074020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.074659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.077802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.127744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 943/1847] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:36 2022] POST /identity/v3/users => generated 292 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:37.133871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.143298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.145429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:37.146801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f17283-0083-4ae2-b563-1a24ee440a06 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.152391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aab557-3204-406e-990b-3ab0e3517f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.153431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aab557-3204-406e-990b-3ab0e3517f55 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.153974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aab557-3204-406e-990b-3ab0e3517f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.154555 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aab557-3204-406e-990b-3ab0e3517f55 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.155385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aab557-3204-406e-990b-3ab0e3517f55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.157243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aab557-3204-406e-990b-3ab0e3517f55 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.159537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552fbbe4-c9c9-4034-9d99-2e611f89f97e admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.165659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.166498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.166886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.167311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] PATH_INFO: `/v3/projects/25f1ebbfb9d042fab037dbe07a651913` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.168747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=25f1ebbfb9d042fab037dbe07a651913)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.171693 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.174967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 905/1848] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:37 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:37.177856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aab557-3204-406e-990b-3ab0e3517f55 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.198827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aab557-3204-406e-990b-3ab0e3517f55 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.207229 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b6be8-7b41-4970-bdba-505a71fae8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.208284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b6be8-7b41-4970-bdba-505a71fae8ae admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.208569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b6be8-7b41-4970-bdba-505a71fae8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.208911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b6be8-7b41-4970-bdba-505a71fae8ae admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.209538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6b6be8-7b41-4970-bdba-505a71fae8ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.211249 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6b6be8-7b41-4970-bdba-505a71fae8ae admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.223296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 906/1849] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:37 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:37.228470 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b6be8-7b41-4970-bdba-505a71fae8ae admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.246530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b6be8-7b41-4970-bdba-505a71fae8ae admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.248853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '25f1ebbfb9d042fab037dbe07a651913', 'request_id': 'req-afa79263-a81d-4c1c-b27b-ac8d07be48d5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:37.258400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '25f1ebbfb9d042fab037dbe07a651913', 'request_id': 'req-afa79263-a81d-4c1c-b27b-ac8d07be48d5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:37.268804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:37.269980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '25f1ebbfb9d042fab037dbe07a651913', 'request_id': 'req-afa79263-a81d-4c1c-b27b-ac8d07be48d5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:37.286957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] The token cache is being invalidate because project 25f1ebbfb9d042fab037dbe07a651913 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:37.287462 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:37.288421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-afa79263-a81d-4c1c-b27b-ac8d07be48d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:37.290439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 944/1850] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:37 2022] DELETE /identity/v3/projects/25f1ebbfb9d042fab037dbe07a651913 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:37.296026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2e32f-d349-43c4-96a4-6756e040f5db None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.297875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2e32f-d349-43c4-96a4-6756e040f5db None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.298155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2e32f-d349-43c4-96a4-6756e040f5db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.320545 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-47c2e32f-d349-43c4-96a4-6756e040f5db None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:37.321756 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-47c2e32f-d349-43c4-96a4-6756e040f5db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:37.322562 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07fdee1-15c6-4a41-b7f2-46fef12359ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.323467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fdee1-15c6-4a41-b7f2-46fef12359ef admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.323830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fdee1-15c6-4a41-b7f2-46fef12359ef admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.324039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fdee1-15c6-4a41-b7f2-46fef12359ef admin admin] PATH_INFO: `/v3/projects/906e8acb9b154d9fad32cfc91b8228c0/users/bb4ca93467fe46d2b7b94f896c017f95/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.340202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 945/1851] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:37.348436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b9a4f3-91c1-44c6-89b6-866a178f7dbd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.348893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b9a4f3-91c1-44c6-89b6-866a178f7dbd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.349262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b9a4f3-91c1-44c6-89b6-866a178f7dbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.354700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07fdee1-15c6-4a41-b7f2-46fef12359ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=906e8acb9b154d9fad32cfc91b8228c0, user_id=bb4ca93467fe46d2b7b94f896c017f95, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.358410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07fdee1-15c6-4a41-b7f2-46fef12359ef admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.374346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-64b9a4f3-91c1-44c6-89b6-866a178f7dbd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:37.375917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-64b9a4f3-91c1-44c6-89b6-866a178f7dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:37.377400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 907/1852] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:37 2022] PUT /identity/v3/projects/906e8acb9b154d9fad32cfc91b8228c0/users/bb4ca93467fe46d2b7b94f896c017f95/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:37.388658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51709477-e6ce-4dfd-b9e7-ddb90614790c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.388983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51709477-e6ce-4dfd-b9e7-ddb90614790c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.389293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51709477-e6ce-4dfd-b9e7-ddb90614790c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.411032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:37.428996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-51709477-e6ce-4dfd-b9e7-ddb90614790c None None] MFA Rules not processed for user `bb4ca93467fe46d2b7b94f896c017f95`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:37.431294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-51709477-e6ce-4dfd-b9e7-ddb90614790c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:37.467151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 946/1853] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:37.471759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b9a4f3-91c1-44c6-89b6-866a178f7dbd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.481368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b9a4f3-91c1-44c6-89b6-866a178f7dbd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.487585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.488254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.488542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.488817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.489503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:37.489812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.491891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.506576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 947/1854] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:52:37.511512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.522663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a104b66-db58-4f06-9fac-47078b2d80f4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.528429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa0a86-97e2-4dd9-b988-f4bd83ba767a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.530088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa0a86-97e2-4dd9-b988-f4bd83ba767a admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.530383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa0a86-97e2-4dd9-b988-f4bd83ba767a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.530663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa0a86-97e2-4dd9-b988-f4bd83ba767a admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.531299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfa0a86-97e2-4dd9-b988-f4bd83ba767a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.532882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 908/1855] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:37.533588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfa0a86-97e2-4dd9-b988-f4bd83ba767a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.537741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51709477-e6ce-4dfd-b9e7-ddb90614790c None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.540294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-51709477-e6ce-4dfd-b9e7-ddb90614790c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:37.568936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51709477-e6ce-4dfd-b9e7-ddb90614790c None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.574961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a900d5-63e3-42dc-9598-0075bb1e8097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.575871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a900d5-63e3-42dc-9598-0075bb1e8097 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.576302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a900d5-63e3-42dc-9598-0075bb1e8097 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.576663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a900d5-63e3-42dc-9598-0075bb1e8097 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.577414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a900d5-63e3-42dc-9598-0075bb1e8097 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.579958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a900d5-63e3-42dc-9598-0075bb1e8097 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.584262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 948/1856] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:37 2022] POST /identity/v3/projects => generated 375 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:37.585876 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa0a86-97e2-4dd9-b988-f4bd83ba767a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.616085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 909/1857] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:37 2022] POST /identity/v3/projects => generated 385 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:37.619944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a900d5-63e3-42dc-9598-0075bb1e8097 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.700466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa0a86-97e2-4dd9-b988-f4bd83ba767a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.707035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb42a1c-1d8e-4193-87dc-f7a0c1178e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.708123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb42a1c-1d8e-4193-87dc-f7a0c1178e4c admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.708638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb42a1c-1d8e-4193-87dc-f7a0c1178e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.709099 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb42a1c-1d8e-4193-87dc-f7a0c1178e4c admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.710296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb42a1c-1d8e-4193-87dc-f7a0c1178e4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.711628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a900d5-63e3-42dc-9598-0075bb1e8097 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.714198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb42a1c-1d8e-4193-87dc-f7a0c1178e4c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.718444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0d213-1d4e-4831-97ee-4c6d1357ea14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.719220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0d213-1d4e-4831-97ee-4c6d1357ea14 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.719519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0d213-1d4e-4831-97ee-4c6d1357ea14 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.719792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0d213-1d4e-4831-97ee-4c6d1357ea14 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.720502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc0d213-1d4e-4831-97ee-4c6d1357ea14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.722883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc0d213-1d4e-4831-97ee-4c6d1357ea14 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.745932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:37.753536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 949/1858] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:37 2022] POST /identity/v3/users => generated 288 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:37.754808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb42a1c-1d8e-4193-87dc-f7a0c1178e4c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.756639 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 910/1859] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:37 2022] POST /identity/v3/users => generated 299 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:37.760947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0d213-1d4e-4831-97ee-4c6d1357ea14 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.766937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb42a1c-1d8e-4193-87dc-f7a0c1178e4c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.773055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76c233e-fd84-4b24-84ce-dd210fd5c967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.773786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c233e-fd84-4b24-84ce-dd210fd5c967 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.774063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c233e-fd84-4b24-84ce-dd210fd5c967 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.774333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c233e-fd84-4b24-84ce-dd210fd5c967 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.775041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76c233e-fd84-4b24-84ce-dd210fd5c967 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.776724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76c233e-fd84-4b24-84ce-dd210fd5c967 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.779231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0d213-1d4e-4831-97ee-4c6d1357ea14 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.784589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8625d3e0-b655-4802-a4d5-be1297f7d8d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.785210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8625d3e0-b655-4802-a4d5-be1297f7d8d1 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.785489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8625d3e0-b655-4802-a4d5-be1297f7d8d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.785763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8625d3e0-b655-4802-a4d5-be1297f7d8d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.786426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8625d3e0-b655-4802-a4d5-be1297f7d8d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.788056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8625d3e0-b655-4802-a4d5-be1297f7d8d1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.795509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 950/1860] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:37 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:37.795509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76c233e-fd84-4b24-84ce-dd210fd5c967 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.800549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 911/1861] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:37 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:37.805135 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8625d3e0-b655-4802-a4d5-be1297f7d8d1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.812109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76c233e-fd84-4b24-84ce-dd210fd5c967 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.816615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8625d3e0-b655-4802-a4d5-be1297f7d8d1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.818031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d9d9ec-c7b9-43d4-b060-c54f1c0a03f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.818649 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d9d9ec-c7b9-43d4-b060-c54f1c0a03f3 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.818930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d9d9ec-c7b9-43d4-b060-c54f1c0a03f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.819204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d9d9ec-c7b9-43d4-b060-c54f1c0a03f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.819847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d9d9ec-c7b9-43d4-b060-c54f1c0a03f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.821490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d9d9ec-c7b9-43d4-b060-c54f1c0a03f3 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.822079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f5aa0a-b1b2-4835-96ad-41c46cb57f7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.822760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f5aa0a-b1b2-4835-96ad-41c46cb57f7a admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.823032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f5aa0a-b1b2-4835-96ad-41c46cb57f7a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.823307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f5aa0a-b1b2-4835-96ad-41c46cb57f7a admin admin] PATH_INFO: `/v3/projects/9563095f340641a0bf389116301a1957/users/96fb72ba503e4a0192f474c0db18d172/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.837357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 951/1862] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:37 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:37.838509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d9d9ec-c7b9-43d4-b060-c54f1c0a03f3 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.842699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f5aa0a-b1b2-4835-96ad-41c46cb57f7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9563095f340641a0bf389116301a1957, user_id=96fb72ba503e4a0192f474c0db18d172, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.845740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f5aa0a-b1b2-4835-96ad-41c46cb57f7a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.849266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d9d9ec-c7b9-43d4-b060-c54f1c0a03f3 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.856041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8159ba89-1b91-474f-b3ef-74379d9b9f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:37.857354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8159ba89-1b91-474f-b3ef-74379d9b9f7e admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.857634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8159ba89-1b91-474f-b3ef-74379d9b9f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.857908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8159ba89-1b91-474f-b3ef-74379d9b9f7e admin admin] PATH_INFO: `/v3/projects/45be9d45ba4c4a2ca1ed053a7263dbee/users/94169e17266b4e6fad7e068a4c1384c7/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.861293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 912/1863] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:37 2022] PUT /identity/v3/projects/9563095f340641a0bf389116301a1957/users/96fb72ba503e4a0192f474c0db18d172/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:37.865469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f5aa0a-b1b2-4835-96ad-41c46cb57f7a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:37.877575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8159ba89-1b91-474f-b3ef-74379d9b9f7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=45be9d45ba4c4a2ca1ed053a7263dbee, user_id=94169e17266b4e6fad7e068a4c1384c7, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:37.880548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8159ba89-1b91-474f-b3ef-74379d9b9f7e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:37.900728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 952/1864] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:37 2022] PUT /identity/v3/projects/45be9d45ba4c4a2ca1ed053a7263dbee/users/94169e17266b4e6fad7e068a4c1384c7/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:37.908768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d49df97-41ab-4e68-9f0c-d348e31ff67b None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:37.909237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d49df97-41ab-4e68-9f0c-d348e31ff67b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:37.909682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d49df97-41ab-4e68-9f0c-d348e31ff67b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:37.934055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-9d49df97-41ab-4e68-9f0c-d348e31ff67b None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:37.934268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f5aa0a-b1b2-4835-96ad-41c46cb57f7a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:37.935301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9d49df97-41ab-4e68-9f0c-d348e31ff67b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:37.971158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:38.009264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7d13ab-7761-440c-a013-d2bd2fd78a50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:38.010039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7d13ab-7761-440c-a013-d2bd2fd78a50 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:38.010465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7d13ab-7761-440c-a013-d2bd2fd78a50 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:38.010828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7d13ab-7761-440c-a013-d2bd2fd78a50 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:38.011631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7d13ab-7761-440c-a013-d2bd2fd78a50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:38.014529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7d13ab-7761-440c-a013-d2bd2fd78a50 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:38.016752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 953/1865] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:38.027585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d49df97-41ab-4e68-9f0c-d348e31ff67b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:38.027787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 913/1866] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:37 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:38.030965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9d49df97-41ab-4e68-9f0c-d348e31ff67b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:38.032744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7d13ab-7761-440c-a013-d2bd2fd78a50 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:38.053106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7d13ab-7761-440c-a013-d2bd2fd78a50 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:38.059787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964ca498-23bc-4901-9e01-3eea6211bd49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:38.060523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ca498-23bc-4901-9e01-3eea6211bd49 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:38.060836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ca498-23bc-4901-9e01-3eea6211bd49 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:38.061183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ca498-23bc-4901-9e01-3eea6211bd49 admin admin] PATH_INFO: `/v3/projects/9563095f340641a0bf389116301a1957/users/96fb72ba503e4a0192f474c0db18d172/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:38.063347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964ca498-23bc-4901-9e01-3eea6211bd49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9563095f340641a0bf389116301a1957, user_id=96fb72ba503e4a0192f474c0db18d172, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:38.066117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964ca498-23bc-4901-9e01-3eea6211bd49 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:38.078308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 914/1867] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:38 2022] PUT /identity/v3/projects/9563095f340641a0bf389116301a1957/users/96fb72ba503e4a0192f474c0db18d172/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:38.090345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3d761f-91ad-44ad-a96e-de5983448929 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:38.090673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3d761f-91ad-44ad-a96e-de5983448929 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:38.090953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3d761f-91ad-44ad-a96e-de5983448929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:38.112787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d49df97-41ab-4e68-9f0c-d348e31ff67b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:38.123492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-0a3d761f-91ad-44ad-a96e-de5983448929 None None] MFA Rules not processed for user `96fb72ba503e4a0192f474c0db18d172`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:38.124887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0a3d761f-91ad-44ad-a96e-de5983448929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:38.170469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ec65c-8d90-4bc1-a111-84da55da5a84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:38.171446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ec65c-8d90-4bc1-a111-84da55da5a84 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:38.171809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ec65c-8d90-4bc1-a111-84da55da5a84 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:38.172191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ec65c-8d90-4bc1-a111-84da55da5a84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:38.185448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:38.218131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 915/1868] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:38 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:38.229468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3d761f-91ad-44ad-a96e-de5983448929 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:38.230974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0a3d761f-91ad-44ad-a96e-de5983448929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:38.241567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327ec65c-8d90-4bc1-a111-84da55da5a84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:38.243018 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327ec65c-8d90-4bc1-a111-84da55da5a84 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:38.254263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 954/1869] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:38.306160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3d761f-91ad-44ad-a96e-de5983448929 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:38.310429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:38.311778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:38.312053 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:38.312358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:38.394724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:38.396323 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:38.409113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 916/1870] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:52:43.417339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6928f17-fd95-4dd7-8b6d-5b88547f697d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:43.417913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6928f17-fd95-4dd7-8b6d-5b88547f697d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:43.417913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6928f17-fd95-4dd7-8b6d-5b88547f697d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:43.459674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a6928f17-fd95-4dd7-8b6d-5b88547f697d None None] MFA Rules not processed for user `94169e17266b4e6fad7e068a4c1384c7`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:43.460928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a6928f17-fd95-4dd7-8b6d-5b88547f697d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:43.501430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:43.547207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 955/1871] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:43.565214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:43.568157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:43.667943 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ebe3af-3472-49c5-bb4b-6067b7f82c83 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:43.673988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:43.674622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:43.674913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:43.675333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:43.684034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:43.686177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:43.702902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 917/1872] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:43 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:45.213736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2827b8-7def-42f3-9ab5-bfaae29ef892 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:45.214267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2827b8-7def-42f3-9ab5-bfaae29ef892 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:45.214411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2827b8-7def-42f3-9ab5-bfaae29ef892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:45.243415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-3d2827b8-7def-42f3-9ab5-bfaae29ef892 None None] MFA Rules not processed for user `94169e17266b4e6fad7e068a4c1384c7`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:45.244845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3d2827b8-7def-42f3-9ab5-bfaae29ef892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:45.258225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 956/1873] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:45 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:47.796872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:47.798638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:47.814201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e73d5-663b-480a-8af9-e2b4c9902670 service nova] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:47.818386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc04ed2-0ab5-4dbb-9f15-8b4c36ad3b43 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:47.819055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc04ed2-0ab5-4dbb-9f15-8b4c36ad3b43 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:47.819337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc04ed2-0ab5-4dbb-9f15-8b4c36ad3b43 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:47.819614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc04ed2-0ab5-4dbb-9f15-8b4c36ad3b43 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:47.830868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc04ed2-0ab5-4dbb-9f15-8b4c36ad3b43 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:47.832372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc04ed2-0ab5-4dbb-9f15-8b4c36ad3b43 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:47.844901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 918/1874] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:47 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:53.067655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2827b8-7def-42f3-9ab5-bfaae29ef892 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:53.077193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2827b8-7def-42f3-9ab5-bfaae29ef892 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:53.082066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:53.082754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:53.083035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:53.083309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] PATH_INFO: `/v3/users/bb4ca93467fe46d2b7b94f896c017f95` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:53.084558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb4ca93467fe46d2b7b94f896c017f95)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:53.086678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:53.142775 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb4ca93467fe46d2b7b94f896c017f95'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.150140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb4ca93467fe46d2b7b94f896c017f95'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.165356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb4ca93467fe46d2b7b94f896c017f95'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.166413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb4ca93467fe46d2b7b94f896c017f95'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.167054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb4ca93467fe46d2b7b94f896c017f95'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.183841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 957/1875] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:53 2022] DELETE /identity/v3/users/bb4ca93467fe46d2b7b94f896c017f95 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:53.188984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e4680-74c2-4632-ad0e-e8f95f645828 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:53.189503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e4680-74c2-4632-ad0e-e8f95f645828 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:53.189635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e4680-74c2-4632-ad0e-e8f95f645828 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:53.204121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:53.211280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e84e4680-74c2-4632-ad0e-e8f95f645828 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:53.212663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e84e4680-74c2-4632-ad0e-e8f95f645828 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:53.275883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 919/1876] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:53.286155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:53.287563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:53.353491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c130d5-fc3d-4d82-b5b1-c16f90bd68f7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:53.357985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ca766f-7350-419e-8fb5-681443125927 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:53.358686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ca766f-7350-419e-8fb5-681443125927 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:53.359033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ca766f-7350-419e-8fb5-681443125927 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:53.359333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ca766f-7350-419e-8fb5-681443125927 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:53.370009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ca766f-7350-419e-8fb5-681443125927 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:53.372342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ca766f-7350-419e-8fb5-681443125927 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:53.382784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 958/1877] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:53.728552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e4680-74c2-4632-ad0e-e8f95f645828 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:53.730030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e84e4680-74c2-4632-ad0e-e8f95f645828 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:53.747888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e4680-74c2-4632-ad0e-e8f95f645828 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:53.752798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:53.753436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:53.753722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:53.753997 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] PATH_INFO: `/v3/projects/906e8acb9b154d9fad32cfc91b8228c0` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:53.755301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=906e8acb9b154d9fad32cfc91b8228c0)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:53.757379 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:53.816546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '906e8acb9b154d9fad32cfc91b8228c0', 'request_id': 'req-3062b75d-3fbc-4268-9000-18c4e843cea2'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.828383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '906e8acb9b154d9fad32cfc91b8228c0', 'request_id': 'req-3062b75d-3fbc-4268-9000-18c4e843cea2'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.830701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '906e8acb9b154d9fad32cfc91b8228c0', 'request_id': 'req-3062b75d-3fbc-4268-9000-18c4e843cea2'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.844623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] The token cache is being invalidate because project 906e8acb9b154d9fad32cfc91b8228c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:53.844914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.845577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.847085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 920/1878] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:53 2022] DELETE /identity/v3/projects/906e8acb9b154d9fad32cfc91b8228c0 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:53.851084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ca766f-7350-419e-8fb5-681443125927 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:53.861358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ca766f-7350-419e-8fb5-681443125927 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:53.900615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:53.921071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:53.921700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:53.921982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:53.922258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] PATH_INFO: `/v3/users/96fb72ba503e4a0192f474c0db18d172` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:53.923368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96fb72ba503e4a0192f474c0db18d172)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:53.925322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:53.975877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96fb72ba503e4a0192f474c0db18d172'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.983569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96fb72ba503e4a0192f474c0db18d172'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.995923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96fb72ba503e4a0192f474c0db18d172'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.996663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96fb72ba503e4a0192f474c0db18d172'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:53.998388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-03a021b5-14b0-4440-8a70-a9975a7b4f23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96fb72ba503e4a0192f474c0db18d172'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:54.013304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 959/1879] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:52:53 2022] DELETE /identity/v3/users/96fb72ba503e4a0192f474c0db18d172 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:54.313038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:54.314530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:54.327549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062b75d-3fbc-4268-9000-18c4e843cea2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:54.399731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:54.400480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:54.400756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:54.401024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] PATH_INFO: `/v3/projects/9563095f340641a0bf389116301a1957` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:54.402170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9563095f340641a0bf389116301a1957)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:54.404978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:54.448535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:54.456776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9563095f340641a0bf389116301a1957', 'request_id': 'req-aca00b4e-fe06-430f-8a49-b9052ea64bd3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:54.463007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9563095f340641a0bf389116301a1957', 'request_id': 'req-aca00b4e-fe06-430f-8a49-b9052ea64bd3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:54.470769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9563095f340641a0bf389116301a1957', 'request_id': 'req-aca00b4e-fe06-430f-8a49-b9052ea64bd3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:54.483772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] The token cache is being invalidate because project 9563095f340641a0bf389116301a1957 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:52:54.484060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:54.484739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aca00b4e-fe06-430f-8a49-b9052ea64bd3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:52:54.486465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 921/1880] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:52:54 2022] DELETE /identity/v3/projects/9563095f340641a0bf389116301a1957 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:54.492483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3a3e06-9b3a-41c9-8bb5-f9e11f901926 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:54.492990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3a3e06-9b3a-41c9-8bb5-f9e11f901926 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:54.493166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3a3e06-9b3a-41c9-8bb5-f9e11f901926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:54.534531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-1c3a3e06-9b3a-41c9-8bb5-f9e11f901926 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:54.536000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1c3a3e06-9b3a-41c9-8bb5-f9e11f901926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:54.634746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 960/1881] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:54.643671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d502cf3-0a67-4571-b2ed-269537566e32 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:54.643956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d502cf3-0a67-4571-b2ed-269537566e32 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:54.644269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d502cf3-0a67-4571-b2ed-269537566e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:54.666903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2d502cf3-0a67-4571-b2ed-269537566e32 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:54.668492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2d502cf3-0a67-4571-b2ed-269537566e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:54.677400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 922/1882] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:54.693745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3a3e06-9b3a-41c9-8bb5-f9e11f901926 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:54.693745 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3a3e06-9b3a-41c9-8bb5-f9e11f901926 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:54.700641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:54.701581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:54.701963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:54.702456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:54.703374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:52:54.703798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:54.705702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:54.722887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 961/1883] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:52:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:52:54.727757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d502cf3-0a67-4571-b2ed-269537566e32 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:54.741406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d502cf3-0a67-4571-b2ed-269537566e32 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:54.747391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f1cf9-bf30-457c-bc8c-e5ee200105c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:54.749299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f1cf9-bf30-457c-bc8c-e5ee200105c0 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:54.749299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f1cf9-bf30-457c-bc8c-e5ee200105c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:54.749299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f1cf9-bf30-457c-bc8c-e5ee200105c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:54.750028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9f1cf9-bf30-457c-bc8c-e5ee200105c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:54.752445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9f1cf9-bf30-457c-bc8c-e5ee200105c0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:54.787159 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 923/1884] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:54 2022] POST /identity/v3/projects => generated 383 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:54.793580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:54.810814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0ae2e7-7e0b-4461-8990-85f464f261ae admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:54.856968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:54.883915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce956211-a4ed-4907-a5f0-31ba78508e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:54.884981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce956211-a4ed-4907-a5f0-31ba78508e92 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:54.885607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce956211-a4ed-4907-a5f0-31ba78508e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:54.886086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce956211-a4ed-4907-a5f0-31ba78508e92 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:54.887442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce956211-a4ed-4907-a5f0-31ba78508e92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:54.892510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce956211-a4ed-4907-a5f0-31ba78508e92 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:54.936981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 962/1885] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:54 2022] POST /identity/v3/users => generated 292 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:54.941193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f1cf9-bf30-457c-bc8c-e5ee200105c0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:54.953095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f1cf9-bf30-457c-bc8c-e5ee200105c0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:54.958061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411617d2-cff8-43a2-8081-df59fdcb270e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:54.958728 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411617d2-cff8-43a2-8081-df59fdcb270e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:54.959003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411617d2-cff8-43a2-8081-df59fdcb270e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:54.959273 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411617d2-cff8-43a2-8081-df59fdcb270e admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:54.959917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411617d2-cff8-43a2-8081-df59fdcb270e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:54.961608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411617d2-cff8-43a2-8081-df59fdcb270e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:54.977548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 924/1886] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:54.979466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce956211-a4ed-4907-a5f0-31ba78508e92 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:54.993994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce956211-a4ed-4907-a5f0-31ba78508e92 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.000361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98d8913-c10c-4c41-9450-0cda24b03a1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.000976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98d8913-c10c-4c41-9450-0cda24b03a1b admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.001256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98d8913-c10c-4c41-9450-0cda24b03a1b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.001539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98d8913-c10c-4c41-9450-0cda24b03a1b admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.002162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98d8913-c10c-4c41-9450-0cda24b03a1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.003747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98d8913-c10c-4c41-9450-0cda24b03a1b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:55.019497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 963/1887] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:54 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:55.023946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411617d2-cff8-43a2-8081-df59fdcb270e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:55.036115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411617d2-cff8-43a2-8081-df59fdcb270e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.041028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26feaeca-5624-4642-9bf6-4fa64b516d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.041766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26feaeca-5624-4642-9bf6-4fa64b516d48 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.042048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26feaeca-5624-4642-9bf6-4fa64b516d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.042325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26feaeca-5624-4642-9bf6-4fa64b516d48 admin admin] PATH_INFO: `/v3/projects/82987a2d87c14fbda7cbb5d84561d277/users/1aa86306891e43038c5fe40edcfd6038/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.060731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26feaeca-5624-4642-9bf6-4fa64b516d48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82987a2d87c14fbda7cbb5d84561d277, user_id=1aa86306891e43038c5fe40edcfd6038, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.063771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26feaeca-5624-4642-9bf6-4fa64b516d48 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:55.080348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 925/1888] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:55 2022] PUT /identity/v3/projects/82987a2d87c14fbda7cbb5d84561d277/users/1aa86306891e43038c5fe40edcfd6038/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:55.095314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48792520-9b50-4533-8262-994bf75686d6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.095314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48792520-9b50-4533-8262-994bf75686d6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.095314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48792520-9b50-4533-8262-994bf75686d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.124181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-48792520-9b50-4533-8262-994bf75686d6 None None] MFA Rules not processed for user `1aa86306891e43038c5fe40edcfd6038`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:55.125423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-48792520-9b50-4533-8262-994bf75686d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:55.202079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 964/1889] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:52:55.206709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26feaeca-5624-4642-9bf6-4fa64b516d48 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:55.271512 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26feaeca-5624-4642-9bf6-4fa64b516d48 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.276191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42556536-2bdc-4799-9eab-f057d175445a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.276799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42556536-2bdc-4799-9eab-f057d175445a admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.277184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42556536-2bdc-4799-9eab-f057d175445a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.277356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42556536-2bdc-4799-9eab-f057d175445a admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.278678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42556536-2bdc-4799-9eab-f057d175445a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.280535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42556536-2bdc-4799-9eab-f057d175445a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:55.314334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 926/1890] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:55 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:55.316196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48792520-9b50-4533-8262-994bf75686d6 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:55.327759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48792520-9b50-4533-8262-994bf75686d6 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.410945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a17b2be-d582-48aa-8627-87868542a747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.411678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a17b2be-d582-48aa-8627-87868542a747 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.411959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a17b2be-d582-48aa-8627-87868542a747 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.412293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a17b2be-d582-48aa-8627-87868542a747 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.412939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a17b2be-d582-48aa-8627-87868542a747 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.414934 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a17b2be-d582-48aa-8627-87868542a747 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:55.442822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 965/1891] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:55 2022] POST /identity/v3/users => generated 293 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:55.450213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42556536-2bdc-4799-9eab-f057d175445a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:55.465235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42556536-2bdc-4799-9eab-f057d175445a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.473263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2f66de-f9d8-4b61-94ba-dfa2dbd4572d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.473738 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2f66de-f9d8-4b61-94ba-dfa2dbd4572d admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.474012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2f66de-f9d8-4b61-94ba-dfa2dbd4572d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.474248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2f66de-f9d8-4b61-94ba-dfa2dbd4572d admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.475499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2f66de-f9d8-4b61-94ba-dfa2dbd4572d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.477340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2f66de-f9d8-4b61-94ba-dfa2dbd4572d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:55.498083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 927/1892] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:55 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:55.501604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a17b2be-d582-48aa-8627-87868542a747 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:55.520292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a17b2be-d582-48aa-8627-87868542a747 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.525812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e361fb14-41f8-4323-aaf3-1cc449002000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.526430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e361fb14-41f8-4323-aaf3-1cc449002000 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.526715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e361fb14-41f8-4323-aaf3-1cc449002000 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.526990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e361fb14-41f8-4323-aaf3-1cc449002000 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.527635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e361fb14-41f8-4323-aaf3-1cc449002000 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.530085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e361fb14-41f8-4323-aaf3-1cc449002000 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:55.545157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 966/1893] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:55 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:55.546379 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2f66de-f9d8-4b61-94ba-dfa2dbd4572d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:55.558719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2f66de-f9d8-4b61-94ba-dfa2dbd4572d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.565516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b4a7b0-ec79-4432-b661-b32e23a82c6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.566177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b4a7b0-ec79-4432-b661-b32e23a82c6f admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.566467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b4a7b0-ec79-4432-b661-b32e23a82c6f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.566743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b4a7b0-ec79-4432-b661-b32e23a82c6f admin admin] PATH_INFO: `/v3/projects/c22cc53f044e46e0b72868e56dc5c505/users/b820d697086443a7abb22e3d87003671/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.591678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b4a7b0-ec79-4432-b661-b32e23a82c6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c22cc53f044e46e0b72868e56dc5c505, user_id=b820d697086443a7abb22e3d87003671, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.595450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b4a7b0-ec79-4432-b661-b32e23a82c6f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:55.623372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 928/1894] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:55 2022] PUT /identity/v3/projects/c22cc53f044e46e0b72868e56dc5c505/users/b820d697086443a7abb22e3d87003671/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:55.627984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9baad75-3e61-475e-8189-b29ccede661e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.628518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9baad75-3e61-475e-8189-b29ccede661e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.628996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9baad75-3e61-475e-8189-b29ccede661e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.667409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f9baad75-3e61-475e-8189-b29ccede661e None None] MFA Rules not processed for user `b820d697086443a7abb22e3d87003671`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:55.668924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f9baad75-3e61-475e-8189-b29ccede661e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:55.698670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:55.756903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 967/1895] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:55.761819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b4a7b0-ec79-4432-b661-b32e23a82c6f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:55.771836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:55.838875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b4a7b0-ec79-4432-b661-b32e23a82c6f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.845429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60666f06-3f42-4e50-99dc-396e2e99da91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.846125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60666f06-3f42-4e50-99dc-396e2e99da91 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.846412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60666f06-3f42-4e50-99dc-396e2e99da91 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.846687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60666f06-3f42-4e50-99dc-396e2e99da91 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.847341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60666f06-3f42-4e50-99dc-396e2e99da91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.849308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60666f06-3f42-4e50-99dc-396e2e99da91 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:55.878309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 929/1896] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:52:55 2022] POST /identity/v3/projects => generated 383 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:52:55.882503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9baad75-3e61-475e-8189-b29ccede661e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:55.895706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9baad75-3e61-475e-8189-b29ccede661e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:55.969106 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3172ace2-8e2a-4a8e-b360-6ae81a122089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:55.970386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3172ace2-8e2a-4a8e-b360-6ae81a122089 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:55.970939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3172ace2-8e2a-4a8e-b360-6ae81a122089 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:55.971450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3172ace2-8e2a-4a8e-b360-6ae81a122089 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:55.972495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3172ace2-8e2a-4a8e-b360-6ae81a122089 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:55.975671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3172ace2-8e2a-4a8e-b360-6ae81a122089 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:56.013895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 968/1897] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:52:55 2022] POST /identity/v3/users => generated 298 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:52:56.018299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60666f06-3f42-4e50-99dc-396e2e99da91 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:56.030343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60666f06-3f42-4e50-99dc-396e2e99da91 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:56.035156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587fd0bd-5e49-4b63-8d71-fca67afbe7ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:56.035758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587fd0bd-5e49-4b63-8d71-fca67afbe7ea admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:56.036066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587fd0bd-5e49-4b63-8d71-fca67afbe7ea admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:56.036366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587fd0bd-5e49-4b63-8d71-fca67afbe7ea admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:56.037011 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587fd0bd-5e49-4b63-8d71-fca67afbe7ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:56.038604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587fd0bd-5e49-4b63-8d71-fca67afbe7ea admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:56.050464 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 930/1898] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:52:56.054667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3172ace2-8e2a-4a8e-b360-6ae81a122089 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:56.064718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3172ace2-8e2a-4a8e-b360-6ae81a122089 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:56.069701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ceac042-be9b-4b65-92af-e36d5b1b6ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:56.070397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceac042-be9b-4b65-92af-e36d5b1b6ed7 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:56.070704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceac042-be9b-4b65-92af-e36d5b1b6ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:56.070985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceac042-be9b-4b65-92af-e36d5b1b6ed7 admin admin] PATH_INFO: `/v3/projects/244c85d644404dd09b5b757fc7110c42/users/196aba6447564b9bbcee514a5d458bb3/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:56.088914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceac042-be9b-4b65-92af-e36d5b1b6ed7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=244c85d644404dd09b5b757fc7110c42, user_id=196aba6447564b9bbcee514a5d458bb3, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:56.092102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceac042-be9b-4b65-92af-e36d5b1b6ed7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:56.106260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 969/1899] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:56 2022] PUT /identity/v3/projects/244c85d644404dd09b5b757fc7110c42/users/196aba6447564b9bbcee514a5d458bb3/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:52:56.110467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587fd0bd-5e49-4b63-8d71-fca67afbe7ea admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:56.120917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587fd0bd-5e49-4b63-8d71-fca67afbe7ea admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:56.180481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8da121f-5d9c-4578-8ee7-b30f6077f713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:56.181091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8da121f-5d9c-4578-8ee7-b30f6077f713 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:56.181376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8da121f-5d9c-4578-8ee7-b30f6077f713 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:56.181754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8da121f-5d9c-4578-8ee7-b30f6077f713 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:56.182395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8da121f-5d9c-4578-8ee7-b30f6077f713 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:56.184005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8da121f-5d9c-4578-8ee7-b30f6077f713 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:56.194723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 931/1900] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:52:56 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:52:56.199255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ceac042-be9b-4b65-92af-e36d5b1b6ed7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:56.210969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ceac042-be9b-4b65-92af-e36d5b1b6ed7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:56.216520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:56.217166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:56.217445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:56.217716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] PATH_INFO: `/v3/projects/244c85d644404dd09b5b757fc7110c42/users/196aba6447564b9bbcee514a5d458bb3/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:56.219945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] RBAC: Authorizing `identity:create_grant(project_id=244c85d644404dd09b5b757fc7110c42, user_id=196aba6447564b9bbcee514a5d458bb3, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:56.222733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:56.234104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 970/1901] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:52:56 2022] PUT /identity/v3/projects/244c85d644404dd09b5b757fc7110c42/users/196aba6447564b9bbcee514a5d458bb3/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:52:56.245163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615a7d94-4265-4846-a766-1f7885528828 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:56.245454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615a7d94-4265-4846-a766-1f7885528828 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:56.245727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615a7d94-4265-4846-a766-1f7885528828 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:56.277651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-615a7d94-4265-4846-a766-1f7885528828 None None] MFA Rules not processed for user `196aba6447564b9bbcee514a5d458bb3`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:52:56.279428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-615a7d94-4265-4846-a766-1f7885528828 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:56.352288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 932/1902] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:52:56 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:52:56.364593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:56.366805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:56.438845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14b519c-c92d-4c42-98b1-1f705593d332 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:56.442857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a671eb74-bffb-4bfb-ad06-86da2c95c1b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:56.443490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a671eb74-bffb-4bfb-ad06-86da2c95c1b5 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:56.443885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a671eb74-bffb-4bfb-ad06-86da2c95c1b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:56.444027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a671eb74-bffb-4bfb-ad06-86da2c95c1b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:56.476534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:56.507471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a671eb74-bffb-4bfb-ad06-86da2c95c1b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:56.508910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a671eb74-bffb-4bfb-ad06-86da2c95c1b5 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:56.518046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 971/1903] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:56 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:58.108435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615a7d94-4265-4846-a766-1f7885528828 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:58.109936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-615a7d94-4265-4846-a766-1f7885528828 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:52:58.117986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:52:58.125525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615a7d94-4265-4846-a766-1f7885528828 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:58.129591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445f65a2-93c5-4517-8a9d-d3c511f387ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:58.130210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445f65a2-93c5-4517-8a9d-d3c511f387ba service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:58.130489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445f65a2-93c5-4517-8a9d-d3c511f387ba service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:58.130759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445f65a2-93c5-4517-8a9d-d3c511f387ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:58.195106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445f65a2-93c5-4517-8a9d-d3c511f387ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:58.196600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445f65a2-93c5-4517-8a9d-d3c511f387ba service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:58.207361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 933/1904] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:58 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:52:59.193536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a671eb74-bffb-4bfb-ad06-86da2c95c1b5 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:52:59.203819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a671eb74-bffb-4bfb-ad06-86da2c95c1b5 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:52:59.207964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed3d05f-1309-410e-9775-0a0ff45f5d12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:52:59.208790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed3d05f-1309-410e-9775-0a0ff45f5d12 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:52:59.209067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed3d05f-1309-410e-9775-0a0ff45f5d12 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:52:59.209338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed3d05f-1309-410e-9775-0a0ff45f5d12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:52:59.220978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed3d05f-1309-410e-9775-0a0ff45f5d12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:52:59.222689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed3d05f-1309-410e-9775-0a0ff45f5d12 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:52:59.234237 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 972/1905] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:52:59 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:53:08.202644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=27 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:09.228555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=20 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:14.993160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445f65a2-93c5-4517-8a9d-d3c511f387ba service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:15.072311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445f65a2-93c5-4517-8a9d-d3c511f387ba service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:15.077784 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:15.078609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:15.078888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:15.079197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] PATH_INFO: `/v3/users/1aa86306891e43038c5fe40edcfd6038` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:15.080519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1aa86306891e43038c5fe40edcfd6038)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:15.082926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:15.142650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1aa86306891e43038c5fe40edcfd6038'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.155879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1aa86306891e43038c5fe40edcfd6038'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.180789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1aa86306891e43038c5fe40edcfd6038'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.181541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1aa86306891e43038c5fe40edcfd6038'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.182576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1aa86306891e43038c5fe40edcfd6038'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.200021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 934/1906] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:53:14 2022] DELETE /identity/v3/users/1aa86306891e43038c5fe40edcfd6038 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:15.204535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401026b-bfdc-4b33-93f2-d5080d07a1c6 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:15.205254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401026b-bfdc-4b33-93f2-d5080d07a1c6 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:15.205254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401026b-bfdc-4b33-93f2-d5080d07a1c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:15.231273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-8401026b-bfdc-4b33-93f2-d5080d07a1c6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:15.232265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8401026b-bfdc-4b33-93f2-d5080d07a1c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:15.321837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 973/1907] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:15.331918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:15.333463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:15.406120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3341cbd5-00f8-467e-939c-783f003ea540 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:15.409882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b59555c-f6fd-4cee-bcc8-ee366832883c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:15.410506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b59555c-f6fd-4cee-bcc8-ee366832883c service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:15.410790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b59555c-f6fd-4cee-bcc8-ee366832883c service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:15.411066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b59555c-f6fd-4cee-bcc8-ee366832883c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:15.421995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b59555c-f6fd-4cee-bcc8-ee366832883c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:15.423442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b59555c-f6fd-4cee-bcc8-ee366832883c service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:15.435877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 935/1908] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:53:15.750636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8401026b-bfdc-4b33-93f2-d5080d07a1c6 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:15.752239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8401026b-bfdc-4b33-93f2-d5080d07a1c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:15.766444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8401026b-bfdc-4b33-93f2-d5080d07a1c6 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:15.771332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:15.771561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:15.772352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:15.772352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] PATH_INFO: `/v3/projects/82987a2d87c14fbda7cbb5d84561d277` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:15.773450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] RBAC: Authorizing `identity:delete_project(project_id=82987a2d87c14fbda7cbb5d84561d277)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:15.775361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:15.841219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82987a2d87c14fbda7cbb5d84561d277', 'request_id': 'req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.848293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82987a2d87c14fbda7cbb5d84561d277', 'request_id': 'req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.858733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82987a2d87c14fbda7cbb5d84561d277', 'request_id': 'req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.873977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] The token cache is being invalidate because project 82987a2d87c14fbda7cbb5d84561d277 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:53:15.874446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.875447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:15.877834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 974/1909] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:53:15 2022] DELETE /identity/v3/projects/82987a2d87c14fbda7cbb5d84561d277 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:15.883870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b59555c-f6fd-4cee-bcc8-ee366832883c service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:15.899723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b59555c-f6fd-4cee-bcc8-ee366832883c service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:15.926264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:15.967945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:15.968616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:15.968896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:15.969177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] PATH_INFO: `/v3/users/b820d697086443a7abb22e3d87003671` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:15.970325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] RBAC: Authorizing `identity:delete_user(user_id=b820d697086443a7abb22e3d87003671)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:15.972299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:16.028103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b820d697086443a7abb22e3d87003671'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.035538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b820d697086443a7abb22e3d87003671'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.048777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b820d697086443a7abb22e3d87003671'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.049579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b820d697086443a7abb22e3d87003671'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.050219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b820d697086443a7abb22e3d87003671'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.071983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 936/1910] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:53:15 2022] DELETE /identity/v3/users/b820d697086443a7abb22e3d87003671 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:16.358722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:16.360182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:16.374590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac8d16d-781c-4f22-9eb4-76c58ad1e66b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:16.437557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:16.438113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:16.438394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:16.438672 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] PATH_INFO: `/v3/projects/c22cc53f044e46e0b72868e56dc5c505` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:16.439804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c22cc53f044e46e0b72868e56dc5c505)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:16.441747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:16.473324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:16.491816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c22cc53f044e46e0b72868e56dc5c505', 'request_id': 'req-575e3a24-5250-4896-9003-b45edba4876a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.498783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c22cc53f044e46e0b72868e56dc5c505', 'request_id': 'req-575e3a24-5250-4896-9003-b45edba4876a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.506254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c22cc53f044e46e0b72868e56dc5c505', 'request_id': 'req-575e3a24-5250-4896-9003-b45edba4876a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.517214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] The token cache is being invalidate because project c22cc53f044e46e0b72868e56dc5c505 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:53:16.517505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.518138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.519626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 975/1911] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:53:16 2022] DELETE /identity/v3/projects/c22cc53f044e46e0b72868e56dc5c505 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:16.523543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:16.524947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:16.538407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2b65d-2003-474d-9259-5dcbd052d97f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:16.599427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:16.600181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:16.600462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:16.600736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] PATH_INFO: `/v3/users/196aba6447564b9bbcee514a5d458bb3` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:16.602264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=196aba6447564b9bbcee514a5d458bb3)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:16.604190 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:16.652145 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:16.655408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '196aba6447564b9bbcee514a5d458bb3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.663625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '196aba6447564b9bbcee514a5d458bb3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.675517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '196aba6447564b9bbcee514a5d458bb3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.676241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '196aba6447564b9bbcee514a5d458bb3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.676880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-e121c509-c0b2-4f21-b413-a039cdd049ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '196aba6447564b9bbcee514a5d458bb3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:16.691365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 937/1912] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:53:16 2022] DELETE /identity/v3/users/196aba6447564b9bbcee514a5d458bb3 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:17.012114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:17.013743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:17.027180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575e3a24-5250-4896-9003-b45edba4876a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:17.095490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:17.096157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.096463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.096735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] PATH_INFO: `/v3/projects/244c85d644404dd09b5b757fc7110c42` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.097924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] RBAC: Authorizing `identity:delete_project(project_id=244c85d644404dd09b5b757fc7110c42)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:17.099839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:17.156740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '244c85d644404dd09b5b757fc7110c42', 'request_id': 'req-7815b1e3-478c-4c27-9d19-deab209cf911'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:17.163432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '244c85d644404dd09b5b757fc7110c42', 'request_id': 'req-7815b1e3-478c-4c27-9d19-deab209cf911'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:17.173604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '244c85d644404dd09b5b757fc7110c42', 'request_id': 'req-7815b1e3-478c-4c27-9d19-deab209cf911'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:17.188395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] The token cache is being invalidate because project 244c85d644404dd09b5b757fc7110c42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:53:17.188933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:17.190034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7815b1e3-478c-4c27-9d19-deab209cf911 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:17.192589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 976/1913] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:53:17 2022] DELETE /identity/v3/projects/244c85d644404dd09b5b757fc7110c42 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:17.201312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3951ac4f-a1e4-41c0-958e-e1126bf72730 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.201893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3951ac4f-a1e4-41c0-958e-e1126bf72730 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.201989 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3951ac4f-a1e4-41c0-958e-e1126bf72730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.223333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3951ac4f-a1e4-41c0-958e-e1126bf72730 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:17.224567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3951ac4f-a1e4-41c0-958e-e1126bf72730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:17.300554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 938/1914] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:53:17.309978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a42305-7ea3-458e-9684-ce91292bb059 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.310267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a42305-7ea3-458e-9684-ce91292bb059 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.310547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a42305-7ea3-458e-9684-ce91292bb059 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.333298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-39a42305-7ea3-458e-9684-ce91292bb059 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:17.334683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-39a42305-7ea3-458e-9684-ce91292bb059 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:17.344684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 977/1915] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:17.350944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3951ac4f-a1e4-41c0-958e-e1126bf72730 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:17.362034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3951ac4f-a1e4-41c0-958e-e1126bf72730 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:17.367189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:17.367850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.368086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.368405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.369051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:53:17.369434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:17.371118 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:17.385331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 939/1916] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:53:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:53:17.391275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a42305-7ea3-458e-9684-ce91292bb059 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:17.407056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a42305-7ea3-458e-9684-ce91292bb059 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:17.414801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ffdbac-7d5a-4157-a204-1353e04e6138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:17.415865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ffdbac-7d5a-4157-a204-1353e04e6138 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.416400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ffdbac-7d5a-4157-a204-1353e04e6138 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.416878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ffdbac-7d5a-4157-a204-1353e04e6138 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.418612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ffdbac-7d5a-4157-a204-1353e04e6138 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:17.422473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ffdbac-7d5a-4157-a204-1353e04e6138 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:17.456638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 978/1917] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:53:17 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:53:17.461677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:17.473042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0784fb2-f1c2-4688-9bf1-0b4934d1b9e2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:17.547999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db598f-3ac5-413f-8821-d589c428de76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:17.549018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db598f-3ac5-413f-8821-d589c428de76 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.549529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db598f-3ac5-413f-8821-d589c428de76 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.550175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db598f-3ac5-413f-8821-d589c428de76 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.551534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3db598f-3ac5-413f-8821-d589c428de76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:17.555878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3db598f-3ac5-413f-8821-d589c428de76 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:17.586431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 940/1918] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:53:17 2022] POST /identity/v3/users => generated 288 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:53:17.592633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ffdbac-7d5a-4157-a204-1353e04e6138 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:17.603821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ffdbac-7d5a-4157-a204-1353e04e6138 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:17.609603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b5ace6-348e-4e01-a3f4-165669c1c9ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:17.610502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b5ace6-348e-4e01-a3f4-165669c1c9ce admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.610807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b5ace6-348e-4e01-a3f4-165669c1c9ce admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.611293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b5ace6-348e-4e01-a3f4-165669c1c9ce admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.611736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b5ace6-348e-4e01-a3f4-165669c1c9ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:17.613644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b5ace6-348e-4e01-a3f4-165669c1c9ce admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:17.627101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 979/1919] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:53:17 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:53:17.630947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db598f-3ac5-413f-8821-d589c428de76 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:17.644581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db598f-3ac5-413f-8821-d589c428de76 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:17.646990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c7f7e5-425f-4119-9e14-1de8fd87843a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:17.647594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7f7e5-425f-4119-9e14-1de8fd87843a admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.647887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7f7e5-425f-4119-9e14-1de8fd87843a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.648229 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7f7e5-425f-4119-9e14-1de8fd87843a admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.648983 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c7f7e5-425f-4119-9e14-1de8fd87843a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:17.650575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c7f7e5-425f-4119-9e14-1de8fd87843a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:17.662804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 941/1920] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:53:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:53:17.667077 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b5ace6-348e-4e01-a3f4-165669c1c9ce admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:17.680365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:17.684357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b5ace6-348e-4e01-a3f4-165669c1c9ce admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:17.691357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:17.692036 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.692356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.692917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] PATH_INFO: `/v3/projects/047bb88ce8ab4fc9b4ce90aa89b3e7c8/users/21e48a878e704c8f8625c1cd00474ec3/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.709498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] RBAC: Authorizing `identity:create_grant(project_id=047bb88ce8ab4fc9b4ce90aa89b3e7c8, user_id=21e48a878e704c8f8625c1cd00474ec3, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:17.712214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:17.726870 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 980/1921] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:53:17 2022] PUT /identity/v3/projects/047bb88ce8ab4fc9b4ce90aa89b3e7c8/users/21e48a878e704c8f8625c1cd00474ec3/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:17.737541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1788c-6e28-436e-b9d4-628a82133fd3 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.737826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1788c-6e28-436e-b9d4-628a82133fd3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.738099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1788c-6e28-436e-b9d4-628a82133fd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.771802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5de1788c-6e28-436e-b9d4-628a82133fd3 None None] MFA Rules not processed for user `21e48a878e704c8f8625c1cd00474ec3`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:17.773080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5de1788c-6e28-436e-b9d4-628a82133fd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:17.787060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:17.849453 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 942/1922] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:17.860984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:17.862726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:17.921903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a73350-b802-4d41-8079-1b6237d8c70e admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:17.925681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f42909-c9ba-4b91-aa83-aa593fd08dba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:17.926298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f42909-c9ba-4b91-aa83-aa593fd08dba service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:17.926576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f42909-c9ba-4b91-aa83-aa593fd08dba service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:17.926846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f42909-c9ba-4b91-aa83-aa593fd08dba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:17.936823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f42909-c9ba-4b91-aa83-aa593fd08dba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:17.938358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f42909-c9ba-4b91-aa83-aa593fd08dba service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:17.948602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 981/1923] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:17 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:53:23.571870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5303096-74c8-4a8c-bb38-a3b9ed21e6c2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:23.572478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5303096-74c8-4a8c-bb38-a3b9ed21e6c2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:23.572591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5303096-74c8-4a8c-bb38-a3b9ed21e6c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:23.591385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b5303096-74c8-4a8c-bb38-a3b9ed21e6c2 None None] MFA Rules not processed for user `21e48a878e704c8f8625c1cd00474ec3`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:23.592745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b5303096-74c8-4a8c-bb38-a3b9ed21e6c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:23.599819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 943/1924] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:24.328830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281dd9a-f005-4602-9339-413a2d5f52fa None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:24.329402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281dd9a-f005-4602-9339-413a2d5f52fa None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:24.329402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281dd9a-f005-4602-9339-413a2d5f52fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:24.361140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-9281dd9a-f005-4602-9339-413a2d5f52fa None None] MFA Rules not processed for user `7127613bde784330bec364f02180291e`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:24.363784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9281dd9a-f005-4602-9339-413a2d5f52fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:24.446415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 982/1925] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:24 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:30.568616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5303096-74c8-4a8c-bb38-a3b9ed21e6c2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:30.570059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b5303096-74c8-4a8c-bb38-a3b9ed21e6c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:30.639671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5303096-74c8-4a8c-bb38-a3b9ed21e6c2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:30.645157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:30.645946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:30.646080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:30.646427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] PATH_INFO: `/v3/users/da5be68da2d84ce0b2434d667869fa8c` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:30.647764 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] RBAC: Authorizing `identity:delete_user(user_id=da5be68da2d84ce0b2434d667869fa8c)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:30.649701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:30.701787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da5be68da2d84ce0b2434d667869fa8c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:30.709125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da5be68da2d84ce0b2434d667869fa8c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:30.723960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da5be68da2d84ce0b2434d667869fa8c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:30.724721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da5be68da2d84ce0b2434d667869fa8c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:30.725328 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da5be68da2d84ce0b2434d667869fa8c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:30.743671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 944/1926] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:53:30 2022] DELETE /identity/v3/users/da5be68da2d84ce0b2434d667869fa8c => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:30.749948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbf92d-bd59-4791-a46f-2168a2166811 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:30.750504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbf92d-bd59-4791-a46f-2168a2166811 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:30.750504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbf92d-bd59-4791-a46f-2168a2166811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:30.769751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a7dbf92d-bd59-4791-a46f-2168a2166811 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:30.770935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a7dbf92d-bd59-4791-a46f-2168a2166811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:30.829509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 983/1927] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:30.839662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:30.841262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:30.874390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:30.901491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec15d9-b830-44f8-a5a1-51d838c9b16d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:30.906308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f92351-e477-450b-8963-1f074d965b46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:30.906915 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f92351-e477-450b-8963-1f074d965b46 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:30.907180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f92351-e477-450b-8963-1f074d965b46 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:30.907448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f92351-e477-450b-8963-1f074d965b46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:30.916359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f92351-e477-450b-8963-1f074d965b46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:30.918918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f92351-e477-450b-8963-1f074d965b46 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:30.928257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 945/1928] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:53:31.343184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dbf92d-bd59-4791-a46f-2168a2166811 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:31.344847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a7dbf92d-bd59-4791-a46f-2168a2166811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:31.356681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dbf92d-bd59-4791-a46f-2168a2166811 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:31.363125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:31.363760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.364037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.364352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] PATH_INFO: `/v3/projects/b676c58e09dc431aafe91d0535cdf2cc` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.365756 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b676c58e09dc431aafe91d0535cdf2cc)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:31.367771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:31.401417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:31.444100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b676c58e09dc431aafe91d0535cdf2cc', 'request_id': 'req-77bf6a78-433d-4ed5-915d-ace876ed7c6b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:31.452285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b676c58e09dc431aafe91d0535cdf2cc', 'request_id': 'req-77bf6a78-433d-4ed5-915d-ace876ed7c6b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:31.466200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b676c58e09dc431aafe91d0535cdf2cc', 'request_id': 'req-77bf6a78-433d-4ed5-915d-ace876ed7c6b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:31.478652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] The token cache is being invalidate because project b676c58e09dc431aafe91d0535cdf2cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:53:31.478951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:31.479604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-77bf6a78-433d-4ed5-915d-ace876ed7c6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:31.481621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 984/1929] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:53:31 2022] DELETE /identity/v3/projects/b676c58e09dc431aafe91d0535cdf2cc => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:31.486740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19289b47-1d4e-4c40-aa60-751d6cc64476 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.487028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19289b47-1d4e-4c40-aa60-751d6cc64476 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.487305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19289b47-1d4e-4c40-aa60-751d6cc64476 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.508685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-19289b47-1d4e-4c40-aa60-751d6cc64476 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:31.510209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-19289b47-1d4e-4c40-aa60-751d6cc64476 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:31.591694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 946/1930] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:31.602271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bffc94-8d0e-48f3-b241-3af35aac09ee None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.602673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bffc94-8d0e-48f3-b241-3af35aac09ee None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.603225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bffc94-8d0e-48f3-b241-3af35aac09ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.631009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-43bffc94-8d0e-48f3-b241-3af35aac09ee None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:31.632757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-43bffc94-8d0e-48f3-b241-3af35aac09ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:31.647828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19289b47-1d4e-4c40-aa60-751d6cc64476 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:31.648754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 985/1931] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:31.660741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19289b47-1d4e-4c40-aa60-751d6cc64476 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:31.666345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:31.667391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.667828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.668697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.669176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:53:31.669482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:31.671239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:31.692333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 947/1932] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:53:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:53:31.693465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bffc94-8d0e-48f3-b241-3af35aac09ee None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:31.708203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bffc94-8d0e-48f3-b241-3af35aac09ee None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:31.714338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:31.716196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.716196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.716456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.717118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:31.719808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:31.763696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 986/1933] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:53:31 2022] POST /identity/v3/projects => generated 373 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:53:31.771154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:31.791305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1a433c-1982-457d-80dc-37d536cc5e75 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:31.806100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:31.808924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:31.885921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ec8962-d020-4c64-b672-187f64785bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:31.887225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ec8962-d020-4c64-b672-187f64785bb5 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.887614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ec8962-d020-4c64-b672-187f64785bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.887996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ec8962-d020-4c64-b672-187f64785bb5 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.888793 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ec8962-d020-4c64-b672-187f64785bb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:31.890870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ec8962-d020-4c64-b672-187f64785bb5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:31.905095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eed525-1cc9-4420-880f-91051200159d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:31.913805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:31.914456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.916245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.916245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] PATH_INFO: `/v3/users/781bb86257534a7980b0ba1f29996b22` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.916887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=781bb86257534a7980b0ba1f29996b22)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:31.919410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:31.921516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 948/1934] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:53:31 2022] POST /identity/v3/users => generated 287 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:53:31.925203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ec8962-d020-4c64-b672-187f64785bb5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:31.937575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ec8962-d020-4c64-b672-187f64785bb5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:31.942609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3112e5a9-e1f4-40d9-a113-ee95374c5050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:31.943291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3112e5a9-e1f4-40d9-a113-ee95374c5050 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.943635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3112e5a9-e1f4-40d9-a113-ee95374c5050 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.943974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3112e5a9-e1f4-40d9-a113-ee95374c5050 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.944705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3112e5a9-e1f4-40d9-a113-ee95374c5050 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:31.946373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3112e5a9-e1f4-40d9-a113-ee95374c5050 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:31.959934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 949/1935] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:53:31.963639 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3112e5a9-e1f4-40d9-a113-ee95374c5050 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:31.980327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3112e5a9-e1f4-40d9-a113-ee95374c5050 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:31.982776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '781bb86257534a7980b0ba1f29996b22'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:31.993815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6f321b-94f3-45c1-aa7d-825d8620dd4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:31.994204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f321b-94f3-45c1-aa7d-825d8620dd4e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:31.994491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f321b-94f3-45c1-aa7d-825d8620dd4e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:31.994706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '781bb86257534a7980b0ba1f29996b22'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:31.994826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f321b-94f3-45c1-aa7d-825d8620dd4e admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:31.995485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6f321b-94f3-45c1-aa7d-825d8620dd4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:31.997286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6f321b-94f3-45c1-aa7d-825d8620dd4e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:32.011731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 950/1936] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:53:32.013004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6f321b-94f3-45c1-aa7d-825d8620dd4e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:32.015682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '781bb86257534a7980b0ba1f29996b22'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:32.016660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '781bb86257534a7980b0ba1f29996b22'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:32.018229 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1aa3f7c9-01c6-4d45-bc27-1190ae85ccd5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '781bb86257534a7980b0ba1f29996b22'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:32.031986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6f321b-94f3-45c1-aa7d-825d8620dd4e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:32.037427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e25e62-7c01-4b11-ab94-89856f99851e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:32.038044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 987/1937] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:53:31 2022] DELETE /identity/v3/users/781bb86257534a7980b0ba1f29996b22 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:32.040660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e25e62-7c01-4b11-ab94-89856f99851e admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.040951 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e25e62-7c01-4b11-ab94-89856f99851e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.043274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31457a19-f42a-459e-8b27-d9e442b8ffdc None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.043562 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31457a19-f42a-459e-8b27-d9e442b8ffdc None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.043838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31457a19-f42a-459e-8b27-d9e442b8ffdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.044838 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e25e62-7c01-4b11-ab94-89856f99851e admin admin] PATH_INFO: `/v3/projects/29e4576f2b744d40878d65470e24c24c/users/82055ff074d842e88e27a441c32e9d01/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.051780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=81,keystone:DELETE=16,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:32.068264 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-31457a19-f42a-459e-8b27-d9e442b8ffdc None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:32.069626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-31457a19-f42a-459e-8b27-d9e442b8ffdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:32.070758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e25e62-7c01-4b11-ab94-89856f99851e admin admin] RBAC: Authorizing `identity:create_grant(project_id=29e4576f2b744d40878d65470e24c24c, user_id=82055ff074d842e88e27a441c32e9d01, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:32.074792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e25e62-7c01-4b11-ab94-89856f99851e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:32.089100 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 951/1938] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:53:32 2022] PUT /identity/v3/projects/29e4576f2b744d40878d65470e24c24c/users/82055ff074d842e88e27a441c32e9d01/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:32.094734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c02f87-7934-4f9b-adb1-25c4a534b3b5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.094989 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c02f87-7934-4f9b-adb1-25c4a534b3b5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.095266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c02f87-7934-4f9b-adb1-25c4a534b3b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.102234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:32.120815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-59c02f87-7934-4f9b-adb1-25c4a534b3b5 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:32.122235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-59c02f87-7934-4f9b-adb1-25c4a534b3b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:32.138021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 988/1939] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:32.147800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31457a19-f42a-459e-8b27-d9e442b8ffdc None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:32.149496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-31457a19-f42a-459e-8b27-d9e442b8ffdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:32.184009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 952/1940] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:32.193769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c02f87-7934-4f9b-adb1-25c4a534b3b5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:32.195160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-59c02f87-7934-4f9b-adb1-25c4a534b3b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:32.208530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31457a19-f42a-459e-8b27-d9e442b8ffdc None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:32.255557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c02f87-7934-4f9b-adb1-25c4a534b3b5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:32.258651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:32.259261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.259539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.259784 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f4bd9c-a8b6-47f9-a1b9-cdc87627573f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:32.259907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.260420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f4bd9c-a8b6-47f9-a1b9-cdc87627573f service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.260704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f4bd9c-a8b6-47f9-a1b9-cdc87627573f service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.261062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f4bd9c-a8b6-47f9-a1b9-cdc87627573f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.269748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:32.270662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f4bd9c-a8b6-47f9-a1b9-cdc87627573f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:32.271159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:32.272073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f4bd9c-a8b6-47f9-a1b9-cdc87627573f service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:32.281217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 989/1941] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:53:32.283470 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 953/1942] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:53:32.579194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:32.581312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:32.604406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6872301d-a26f-43d9-90a4-6309643e6ff2 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:32.614403 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:32.616005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.616704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.617408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] PATH_INFO: `/v3/projects/c6f967be521f4de6afbddd1eae7ad283` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.621194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6f967be521f4de6afbddd1eae7ad283)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:32.625221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:32.702495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6f967be521f4de6afbddd1eae7ad283', 'request_id': 'req-4cad9749-e69b-4437-8273-95230faf00f1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:32.710781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6f967be521f4de6afbddd1eae7ad283', 'request_id': 'req-4cad9749-e69b-4437-8273-95230faf00f1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:32.715866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:32.721056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6f967be521f4de6afbddd1eae7ad283', 'request_id': 'req-4cad9749-e69b-4437-8273-95230faf00f1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:32.742282 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] The token cache is being invalidate because project c6f967be521f4de6afbddd1eae7ad283 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:53:32.742664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:32.743310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4cad9749-e69b-4437-8273-95230faf00f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:32.745097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 990/1943] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:53:32 2022] DELETE /identity/v3/projects/c6f967be521f4de6afbddd1eae7ad283 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:32.750701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dd3609-4db2-4f69-a905-6eb8aaa745dd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.751705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dd3609-4db2-4f69-a905-6eb8aaa745dd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.752108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dd3609-4db2-4f69-a905-6eb8aaa745dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.780201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-10dd3609-4db2-4f69-a905-6eb8aaa745dd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:32.781832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-10dd3609-4db2-4f69-a905-6eb8aaa745dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:32.860268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 954/1944] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:32.869635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9d2d86-a621-4e82-b2d5-05ee37a10ac5 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.870152 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9d2d86-a621-4e82-b2d5-05ee37a10ac5 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.870824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9d2d86-a621-4e82-b2d5-05ee37a10ac5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.894121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ed9d2d86-a621-4e82-b2d5-05ee37a10ac5 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:32.895260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ed9d2d86-a621-4e82-b2d5-05ee37a10ac5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:32.904274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 991/1945] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:32.908728 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dd3609-4db2-4f69-a905-6eb8aaa745dd None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:32.919743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dd3609-4db2-4f69-a905-6eb8aaa745dd None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:32.925433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:32.926057 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.926339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.926615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.927277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:53:32.927584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:32.929500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:32.943836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 955/1946] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:53:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:53:32.950094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9d2d86-a621-4e82-b2d5-05ee37a10ac5 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:32.965896 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9d2d86-a621-4e82-b2d5-05ee37a10ac5 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:32.997077 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b90753b-89cf-45ca-83f6-c728150d23be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:32.997778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90753b-89cf-45ca-83f6-c728150d23be admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:32.998067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90753b-89cf-45ca-83f6-c728150d23be admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:32.998345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90753b-89cf-45ca-83f6-c728150d23be admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:32.999090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b90753b-89cf-45ca-83f6-c728150d23be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:33.001027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b90753b-89cf-45ca-83f6-c728150d23be admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:33.039876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:33.044268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 992/1947] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:53:32 2022] POST /identity/v3/projects => generated 371 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:53:33.054356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9667b23-0480-451c-beef-5b4bc1261050 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:33.099673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:33.145445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c3fd1-20a2-4cb1-86db-a2d506dba9e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:33.146071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c3fd1-20a2-4cb1-86db-a2d506dba9e2 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:33.147410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c3fd1-20a2-4cb1-86db-a2d506dba9e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:33.147410 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c3fd1-20a2-4cb1-86db-a2d506dba9e2 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:33.147727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44c3fd1-20a2-4cb1-86db-a2d506dba9e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:33.149722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44c3fd1-20a2-4cb1-86db-a2d506dba9e2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:33.178703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 956/1948] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:53:33 2022] POST /identity/v3/users => generated 286 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:53:33.182756 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b90753b-89cf-45ca-83f6-c728150d23be admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:33.195598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b90753b-89cf-45ca-83f6-c728150d23be admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:33.201822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07857ce0-6bca-44df-b950-fecb44da202d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:33.202601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07857ce0-6bca-44df-b950-fecb44da202d admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:33.203093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07857ce0-6bca-44df-b950-fecb44da202d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:33.203431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07857ce0-6bca-44df-b950-fecb44da202d admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:33.204329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07857ce0-6bca-44df-b950-fecb44da202d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:33.206143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07857ce0-6bca-44df-b950-fecb44da202d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:33.223256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 993/1949] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:53:33 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:53:33.227431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c3fd1-20a2-4cb1-86db-a2d506dba9e2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:33.238513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c3fd1-20a2-4cb1-86db-a2d506dba9e2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:33.245604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6901ce2-042c-49c1-83f0-697d742fc6c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:33.245982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6901ce2-042c-49c1-83f0-697d742fc6c4 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:33.246318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6901ce2-042c-49c1-83f0-697d742fc6c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:33.246619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6901ce2-042c-49c1-83f0-697d742fc6c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:33.247285 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6901ce2-042c-49c1-83f0-697d742fc6c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:33.249191 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6901ce2-042c-49c1-83f0-697d742fc6c4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:33.266917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07857ce0-6bca-44df-b950-fecb44da202d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:33.267359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 957/1950] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:53:33 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:53:33.278210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07857ce0-6bca-44df-b950-fecb44da202d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:33.284673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:33.285369 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:33.285655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:33.286304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] PATH_INFO: `/v3/projects/4c691ad8428c478998a7cfef57480f67/users/5de7f199421540b0904cb85c705f9bb5/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:33.304982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c691ad8428c478998a7cfef57480f67, user_id=5de7f199421540b0904cb85c705f9bb5, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:33.308087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:33.326069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 994/1951] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:53:33 2022] PUT /identity/v3/projects/4c691ad8428c478998a7cfef57480f67/users/5de7f199421540b0904cb85c705f9bb5/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:33.337624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbbdc82-be34-4fd4-baa2-254e8a5425e5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:33.337913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbbdc82-be34-4fd4-baa2-254e8a5425e5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:33.338189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbbdc82-be34-4fd4-baa2-254e8a5425e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:33.377048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-0bbbdc82-be34-4fd4-baa2-254e8a5425e5 None None] MFA Rules not processed for user `5de7f199421540b0904cb85c705f9bb5`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:33.378489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0bbbdc82-be34-4fd4-baa2-254e8a5425e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:33.469883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 958/1952] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:33 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:33.481998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:33.483779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:33.560207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc6b6e4-d1bf-4e43-ba01-f581009139ca admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:33.565254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11953e6-2828-4d46-9f6e-432bc25e529e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:33.566182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11953e6-2828-4d46-9f6e-432bc25e529e service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:33.566583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11953e6-2828-4d46-9f6e-432bc25e529e service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:33.566928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11953e6-2828-4d46-9f6e-432bc25e529e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:33.576555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11953e6-2828-4d46-9f6e-432bc25e529e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:33.578110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11953e6-2828-4d46-9f6e-432bc25e529e service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:33.588532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 995/1953] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:33 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:53:33.617253 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbbdc82-be34-4fd4-baa2-254e8a5425e5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:33.705984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbbdc82-be34-4fd4-baa2-254e8a5425e5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:33.712520 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:33.713261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:33.713602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:33.713877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] PATH_INFO: `/v3/users/5de7f199421540b0904cb85c705f9bb5` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:33.715953 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=5de7f199421540b0904cb85c705f9bb5)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:33.718022 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:33.778884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:33.786649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5de7f199421540b0904cb85c705f9bb5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:33.797385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5de7f199421540b0904cb85c705f9bb5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:33.817996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5de7f199421540b0904cb85c705f9bb5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:33.818757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5de7f199421540b0904cb85c705f9bb5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:33.819387 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5de7f199421540b0904cb85c705f9bb5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:33.839719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 959/1954] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:53:33 2022] DELETE /identity/v3/users/5de7f199421540b0904cb85c705f9bb5 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:33.844588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a580f9a7-4e1d-4bc3-a97a-9efc7a693759 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:33.844862 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a580f9a7-4e1d-4bc3-a97a-9efc7a693759 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:33.845119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a580f9a7-4e1d-4bc3-a97a-9efc7a693759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:33.867787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a580f9a7-4e1d-4bc3-a97a-9efc7a693759 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:33.869038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a580f9a7-4e1d-4bc3-a97a-9efc7a693759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:33.945397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 996/1955] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:33.956642 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:33.958491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:34.119365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47acd59e-2dc0-43ab-aa47-cb51d64d82ca admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:34.131259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552f70d2-5d0e-4714-bb55-3670c9c41bbc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:34.138984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552f70d2-5d0e-4714-bb55-3670c9c41bbc service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:34.139412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552f70d2-5d0e-4714-bb55-3670c9c41bbc service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:34.139698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552f70d2-5d0e-4714-bb55-3670c9c41bbc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:34.168608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552f70d2-5d0e-4714-bb55-3670c9c41bbc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:34.172802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552f70d2-5d0e-4714-bb55-3670c9c41bbc service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:34.206645 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 960/1956] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:53:34.726087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a580f9a7-4e1d-4bc3-a97a-9efc7a693759 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:34.728039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a580f9a7-4e1d-4bc3-a97a-9efc7a693759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:34.745461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a580f9a7-4e1d-4bc3-a97a-9efc7a693759 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:34.752274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:34.753321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:34.753898 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:34.754359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] PATH_INFO: `/v3/projects/4c691ad8428c478998a7cfef57480f67` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:34.755931 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c691ad8428c478998a7cfef57480f67)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:34.758890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:34.785155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:34.817748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c691ad8428c478998a7cfef57480f67', 'request_id': 'req-3bec9e04-c509-4fda-9b0e-1a88b754f481'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:34.826175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c691ad8428c478998a7cfef57480f67', 'request_id': 'req-3bec9e04-c509-4fda-9b0e-1a88b754f481'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:34.835126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c691ad8428c478998a7cfef57480f67', 'request_id': 'req-3bec9e04-c509-4fda-9b0e-1a88b754f481'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:34.847524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] The token cache is being invalidate because project 4c691ad8428c478998a7cfef57480f67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:53:34.847932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:34.848684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:34.850494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 997/1957] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:53:34 2022] DELETE /identity/v3/projects/4c691ad8428c478998a7cfef57480f67 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:37.401380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b00c04a-0466-497f-86da-6d95bef13ebd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:37.401896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b00c04a-0466-497f-86da-6d95bef13ebd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:37.402048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b00c04a-0466-497f-86da-6d95bef13ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:37.431307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7b00c04a-0466-497f-86da-6d95bef13ebd None None] MFA Rules not processed for user `82055ff074d842e88e27a441c32e9d01`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:37.433362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7b00c04a-0466-497f-86da-6d95bef13ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:37.503183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 961/1958] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:37 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:37.507744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:37.509242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:37.577678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bec9e04-c509-4fda-9b0e-1a88b754f481 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:37.582448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c244da-15ef-4479-9b8a-7c9527251167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:37.583057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c244da-15ef-4479-9b8a-7c9527251167 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:37.583337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c244da-15ef-4479-9b8a-7c9527251167 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:37.583628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c244da-15ef-4479-9b8a-7c9527251167 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:37.584343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c244da-15ef-4479-9b8a-7c9527251167 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:37.586327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c244da-15ef-4479-9b8a-7c9527251167 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:37.619613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 998/1959] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:53:37 2022] POST /identity/v3/projects => generated 371 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:53:37.623604 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b00c04a-0466-497f-86da-6d95bef13ebd None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:37.634412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b00c04a-0466-497f-86da-6d95bef13ebd None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:37.685742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:37.697510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d49ce-c295-4507-ae0b-658652276f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:37.698138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d49ce-c295-4507-ae0b-658652276f3e admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:37.698632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d49ce-c295-4507-ae0b-658652276f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:37.698906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d49ce-c295-4507-ae0b-658652276f3e admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:37.699566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372d49ce-c295-4507-ae0b-658652276f3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:37.701623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372d49ce-c295-4507-ae0b-658652276f3e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:37.724928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 962/1960] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:53:37 2022] POST /identity/v3/users => generated 292 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:53:37.728750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c244da-15ef-4479-9b8a-7c9527251167 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:37.740457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c244da-15ef-4479-9b8a-7c9527251167 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:37.745247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c1543-50d7-4a2d-86d4-c8102692009b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:37.746038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c1543-50d7-4a2d-86d4-c8102692009b admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:37.746386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c1543-50d7-4a2d-86d4-c8102692009b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:37.746731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c1543-50d7-4a2d-86d4-c8102692009b admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:37.747443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6c1543-50d7-4a2d-86d4-c8102692009b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:37.749076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6c1543-50d7-4a2d-86d4-c8102692009b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:37.759992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 999/1961] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:53:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:53:37.764889 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d49ce-c295-4507-ae0b-658652276f3e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:37.779516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d49ce-c295-4507-ae0b-658652276f3e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:37.782766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230c7dac-1088-4aa9-ad44-4f253d5caf3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:37.783728 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c7dac-1088-4aa9-ad44-4f253d5caf3b admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:37.783856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c7dac-1088-4aa9-ad44-4f253d5caf3b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:37.785805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c7dac-1088-4aa9-ad44-4f253d5caf3b admin admin] PATH_INFO: `/v3/projects/0c91fbd0e15a466a894e90236b7c0f90/users/dbbaeca00feb411b84c495b267a618e8/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:37.802519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230c7dac-1088-4aa9-ad44-4f253d5caf3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c91fbd0e15a466a894e90236b7c0f90, user_id=dbbaeca00feb411b84c495b267a618e8, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:37.805218 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230c7dac-1088-4aa9-ad44-4f253d5caf3b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:37.819239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 963/1962] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:53:37 2022] PUT /identity/v3/projects/0c91fbd0e15a466a894e90236b7c0f90/users/dbbaeca00feb411b84c495b267a618e8/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:37.824256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c1543-50d7-4a2d-86d4-c8102692009b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:37.834371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c1543-50d7-4a2d-86d4-c8102692009b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:37.906507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a2db54-18d0-4a91-8caf-8360c1edbc4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:37.907219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2db54-18d0-4a91-8caf-8360c1edbc4c admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:37.907592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2db54-18d0-4a91-8caf-8360c1edbc4c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:37.908093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2db54-18d0-4a91-8caf-8360c1edbc4c admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:37.909358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a2db54-18d0-4a91-8caf-8360c1edbc4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:37.911119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a2db54-18d0-4a91-8caf-8360c1edbc4c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:37.926959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1000/1963] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:53:37 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:53:37.928431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230c7dac-1088-4aa9-ad44-4f253d5caf3b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:37.940124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230c7dac-1088-4aa9-ad44-4f253d5caf3b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:37.945421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:37.945997 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:37.946282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:37.946557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] PATH_INFO: `/v3/projects/0c91fbd0e15a466a894e90236b7c0f90/users/dbbaeca00feb411b84c495b267a618e8/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:37.949445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c91fbd0e15a466a894e90236b7c0f90, user_id=dbbaeca00feb411b84c495b267a618e8, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:37.951949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:37.965264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 964/1964] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:53:37 2022] PUT /identity/v3/projects/0c91fbd0e15a466a894e90236b7c0f90/users/dbbaeca00feb411b84c495b267a618e8/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:53:43.653691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347cb665-6784-4023-b3cb-4707a6808701 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:43.654291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347cb665-6784-4023-b3cb-4707a6808701 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:43.654291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347cb665-6784-4023-b3cb-4707a6808701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:43.699254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-347cb665-6784-4023-b3cb-4707a6808701 None None] MFA Rules not processed for user `dbbaeca00feb411b84c495b267a618e8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:43.700539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-347cb665-6784-4023-b3cb-4707a6808701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:43.716276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:43.837020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1001/1965] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:43 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:43.851160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:43.853028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:44.009182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4035742f-5eac-42cc-97ec-ca43df933b56 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:44.017737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305edabf-95b4-4bdb-9a35-89efb52e4770 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:44.018308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305edabf-95b4-4bdb-9a35-89efb52e4770 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:44.018588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305edabf-95b4-4bdb-9a35-89efb52e4770 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:44.018857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305edabf-95b4-4bdb-9a35-89efb52e4770 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:44.085162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305edabf-95b4-4bdb-9a35-89efb52e4770 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:44.086893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305edabf-95b4-4bdb-9a35-89efb52e4770 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:44.098297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 965/1966] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:43 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:53:45.702823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c20a94-eaec-40a4-8b9d-d4a1a1ccdafd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:45.703274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c20a94-eaec-40a4-8b9d-d4a1a1ccdafd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:45.703658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c20a94-eaec-40a4-8b9d-d4a1a1ccdafd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:45.709976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e247f-61ec-4fe1-bbd3-f9cc2e18bb68 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:45.710432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e247f-61ec-4fe1-bbd3-f9cc2e18bb68 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:45.710680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e247f-61ec-4fe1-bbd3-f9cc2e18bb68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:45.732098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-322e247f-61ec-4fe1-bbd3-f9cc2e18bb68 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:45.732821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-23c20a94-eaec-40a4-8b9d-d4a1a1ccdafd None None] MFA Rules not processed for user `82055ff074d842e88e27a441c32e9d01`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:45.733967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-322e247f-61ec-4fe1-bbd3-f9cc2e18bb68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:45.734105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-23c20a94-eaec-40a4-8b9d-d4a1a1ccdafd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:45.752880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1002/1967] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:45.796448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:45.819685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 966/1968] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:53:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:53:45.900163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c20a94-eaec-40a4-8b9d-d4a1a1ccdafd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:45.901826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-23c20a94-eaec-40a4-8b9d-d4a1a1ccdafd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:46.015071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c20a94-eaec-40a4-8b9d-d4a1a1ccdafd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:46.030374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48a4e16-e93c-4f96-a228-60dabf5f8aac service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:46.031004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48a4e16-e93c-4f96-a228-60dabf5f8aac service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:46.031287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48a4e16-e93c-4f96-a228-60dabf5f8aac service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:46.031560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48a4e16-e93c-4f96-a228-60dabf5f8aac service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:46.043400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48a4e16-e93c-4f96-a228-60dabf5f8aac service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:46.047427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48a4e16-e93c-4f96-a228-60dabf5f8aac service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:46.058189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1003/1969] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:53:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:53:46.837955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb9921-f912-47d6-badf-9344c900c02a None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:46.838472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb9921-f912-47d6-badf-9344c900c02a None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:46.838719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb9921-f912-47d6-badf-9344c900c02a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:46.883016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-33cb9921-f912-47d6-badf-9344c900c02a None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:46.884307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-33cb9921-f912-47d6-badf-9344c900c02a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:46.900363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 967/1970] 10.222.0.39 () {58 vars in 967 bytes} [Mon Oct 24 08:53:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:53:46.917954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48a4e16-e93c-4f96-a228-60dabf5f8aac service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:46.928653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48a4e16-e93c-4f96-a228-60dabf5f8aac service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:46.937204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292bd647-4a03-482a-aa0f-d121fcdfb217 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:46.937900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292bd647-4a03-482a-aa0f-d121fcdfb217 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:46.938409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292bd647-4a03-482a-aa0f-d121fcdfb217 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:46.938682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292bd647-4a03-482a-aa0f-d121fcdfb217 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:46.950987 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292bd647-4a03-482a-aa0f-d121fcdfb217 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:46.953033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292bd647-4a03-482a-aa0f-d121fcdfb217 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:46.963956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1004/1971] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:53:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:53:47.306750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf3d8a1-ed06-4371-953c-0fd138fda5df None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:47.307829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf3d8a1-ed06-4371-953c-0fd138fda5df None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:47.308291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf3d8a1-ed06-4371-953c-0fd138fda5df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:47.331183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3cf3d8a1-ed06-4371-953c-0fd138fda5df None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:47.332671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3cf3d8a1-ed06-4371-953c-0fd138fda5df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:47.342628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 968/1972] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:53:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:47.366119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292bd647-4a03-482a-aa0f-d121fcdfb217 service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:47.387618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292bd647-4a03-482a-aa0f-d121fcdfb217 service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:47.394263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f99e246-a03a-41d7-aa79-1213d5262a87 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:47.395188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99e246-a03a-41d7-aa79-1213d5262a87 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:47.395660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99e246-a03a-41d7-aa79-1213d5262a87 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:47.396170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99e246-a03a-41d7-aa79-1213d5262a87 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:47.406818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f99e246-a03a-41d7-aa79-1213d5262a87 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:47.408772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f99e246-a03a-41d7-aa79-1213d5262a87 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:47.419180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1005/1973] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:53:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:53:47.808624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd1a68-6019-41d4-b4a8-d5f52032acbf None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:47.809095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd1a68-6019-41d4-b4a8-d5f52032acbf None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:47.809377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd1a68-6019-41d4-b4a8-d5f52032acbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:47.839958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b2cd1a68-6019-41d4-b4a8-d5f52032acbf None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:47.842073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b2cd1a68-6019-41d4-b4a8-d5f52032acbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:47.849465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 969/1974] 10.222.0.39 () {58 vars in 967 bytes} [Mon Oct 24 08:53:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:47.866818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f99e246-a03a-41d7-aa79-1213d5262a87 service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:47.879894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f99e246-a03a-41d7-aa79-1213d5262a87 service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:47.885207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5de060-a4e9-4e82-86dd-758cc32f18b6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:47.886573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5de060-a4e9-4e82-86dd-758cc32f18b6 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:47.886919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5de060-a4e9-4e82-86dd-758cc32f18b6 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:47.887258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5de060-a4e9-4e82-86dd-758cc32f18b6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:47.895715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5de060-a4e9-4e82-86dd-758cc32f18b6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:47.897359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5de060-a4e9-4e82-86dd-758cc32f18b6 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:47.907343 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1006/1975] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:53:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:53:50.158833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cd1a68-6019-41d4-b4a8-d5f52032acbf None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:50.160281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b2cd1a68-6019-41d4-b4a8-d5f52032acbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:50.232088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cd1a68-6019-41d4-b4a8-d5f52032acbf None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:50.236804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:50.237664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:50.237751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:50.237979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] PATH_INFO: `/v3/users/21e48a878e704c8f8625c1cd00474ec3` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:50.239474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] RBAC: Authorizing `identity:delete_user(user_id=21e48a878e704c8f8625c1cd00474ec3)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:50.241650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:50.292629 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21e48a878e704c8f8625c1cd00474ec3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.299515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21e48a878e704c8f8625c1cd00474ec3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.312395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21e48a878e704c8f8625c1cd00474ec3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.313244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21e48a878e704c8f8625c1cd00474ec3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.313966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21e48a878e704c8f8625c1cd00474ec3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.328903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 970/1976] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:53:50 2022] DELETE /identity/v3/users/21e48a878e704c8f8625c1cd00474ec3 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:53:50.333430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7c887-c8a3-476a-a880-de8f3cb8b274 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:50.333887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7c887-c8a3-476a-a880-de8f3cb8b274 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:50.334142 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7c887-c8a3-476a-a880-de8f3cb8b274 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:50.353689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-efe7c887-c8a3-476a-a880-de8f3cb8b274 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:53:50.355043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-efe7c887-c8a3-476a-a880-de8f3cb8b274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:50.406632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:50.424937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1007/1977] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:53:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:53:50.435877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:50.437350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:50.490132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:53:50.498306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0208f7dc-c026-4094-9c7c-93ecaff49b0a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:50.503413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:50.504050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:50.504520 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:50.504842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:50.514847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:50.516314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:50.526039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 971/1978] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:53:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:53:50.714770 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe7c887-c8a3-476a-a880-de8f3cb8b274 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:53:50.716823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-efe7c887-c8a3-476a-a880-de8f3cb8b274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:53:50.729444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe7c887-c8a3-476a-a880-de8f3cb8b274 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:53:50.745272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:53:50.745911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:53:50.746188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:53:50.746463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] PATH_INFO: `/v3/projects/047bb88ce8ab4fc9b4ce90aa89b3e7c8` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:53:50.747761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=047bb88ce8ab4fc9b4ce90aa89b3e7c8)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:53:50.749808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:53:50.798047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '047bb88ce8ab4fc9b4ce90aa89b3e7c8', 'request_id': 'req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.810335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '047bb88ce8ab4fc9b4ce90aa89b3e7c8', 'request_id': 'req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.819321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '047bb88ce8ab4fc9b4ce90aa89b3e7c8', 'request_id': 'req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.831566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] The token cache is being invalidate because project 047bb88ce8ab4fc9b4ce90aa89b3e7c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:53:50.832064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.832818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:53:50.834544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1008/1979] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:53:50 2022] DELETE /identity/v3/projects/047bb88ce8ab4fc9b4ce90aa89b3e7c8 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:54:00.521689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:54:00.828193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=33,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:54:11.086715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:54:11.088906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:11.164512 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a0159-ff45-45ad-a3f5-b6df2460e68f service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:54:11.230564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74685dc-9685-498a-b712-dc8d5fe4d45b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:54:11.231175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74685dc-9685-498a-b712-dc8d5fe4d45b service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:11.231457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74685dc-9685-498a-b712-dc8d5fe4d45b service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:11.231724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74685dc-9685-498a-b712-dc8d5fe4d45b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:11.233558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d74685dc-9685-498a-b712-dc8d5fe4d45b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:11.247754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74685dc-9685-498a-b712-dc8d5fe4d45b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:54:11.249501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74685dc-9685-498a-b712-dc8d5fe4d45b service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:54:11.261544 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 972/1980] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:54:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:54:21.255757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=58 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:54:29.792073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:54:29.793663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:29.857142 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cca089-dd1e-4a7d-8985-7f38fcbf39a3 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:54:29.860728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:54:29.861332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:29.861613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:29.861888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:29.863628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:29.928173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:54:29.929535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:54:29.939658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1009/1981] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:54:29 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:54:36.214640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232bb43a-47e8-4332-b7a0-319b6b65dae2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:36.215150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232bb43a-47e8-4332-b7a0-319b6b65dae2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:36.215257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232bb43a-47e8-4332-b7a0-319b6b65dae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:36.235210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-232bb43a-47e8-4332-b7a0-319b6b65dae2 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:54:36.236438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-232bb43a-47e8-4332-b7a0-319b6b65dae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:36.293921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 973/1982] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:54:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:54:36.306172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:54:36.307793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:36.376561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f98518d-4806-45d4-80b3-a99a307987dc service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:54:36.382329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad59e414-33af-46b3-91be-57491d120540 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:54:36.382924 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad59e414-33af-46b3-91be-57491d120540 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:36.383200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad59e414-33af-46b3-91be-57491d120540 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:36.383465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad59e414-33af-46b3-91be-57491d120540 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:36.391912 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad59e414-33af-46b3-91be-57491d120540 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:54:36.393504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad59e414-33af-46b3-91be-57491d120540 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:54:36.402025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1010/1983] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:54:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 24 08:54:43.870235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea6de5-7141-443b-ab02-84b0e2df056f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:43.870774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea6de5-7141-443b-ab02-84b0e2df056f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:43.870884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea6de5-7141-443b-ab02-84b0e2df056f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:43.895384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7aea6de5-7141-443b-ab02-84b0e2df056f None None] MFA Rules not processed for user `ebdef1d69bb14837b9a0c28cbba332fc`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:54:43.897010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7aea6de5-7141-443b-ab02-84b0e2df056f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:43.972317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 974/1984] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:54:43 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:54:46.398101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=82 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:54:52.303572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a336b87-4aa7-47ad-afc4-972f46f9b16b None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.304245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a336b87-4aa7-47ad-afc4-972f46f9b16b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.304245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a336b87-4aa7-47ad-afc4-972f46f9b16b None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.307203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1011/1985] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:54:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:54:52.322202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10eb00c-1444-458c-9548-39610f969853 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.322863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10eb00c-1444-458c-9548-39610f969853 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.322977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10eb00c-1444-458c-9548-39610f969853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.343594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a10eb00c-1444-458c-9548-39610f969853 None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:54:52.344857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a10eb00c-1444-458c-9548-39610f969853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:52.417454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 975/1986] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:54:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:54:52.429820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a336b87-4aa7-47ad-afc4-972f46f9b16b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:54:52.431786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6a336b87-4aa7-47ad-afc4-972f46f9b16b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:52.492084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a336b87-4aa7-47ad-afc4-972f46f9b16b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:54:52.497308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b62b5e-08d2-4f08-9d2b-b75cf608038d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a4fd11d5d85479c9aa1d87b23a13141', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:54:52.498019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b62b5e-08d2-4f08-9d2b-b75cf608038d service glance] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.498309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b62b5e-08d2-4f08-9d2b-b75cf608038d service glance] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.498606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b62b5e-08d2-4f08-9d2b-b75cf608038d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.509524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b62b5e-08d2-4f08-9d2b-b75cf608038d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:54:52.510951 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b62b5e-08d2-4f08-9d2b-b75cf608038d service glance] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:54:52.521983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1012/1987] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:54:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:54:52.593834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7f7354-86b9-4e72-87bb-6b9a6b4fd56f None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.594017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7f7354-86b9-4e72-87bb-6b9a6b4fd56f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.594322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7f7354-86b9-4e72-87bb-6b9a6b4fd56f None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.595659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 976/1988] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:54:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:54:52.600420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6365e89-f097-49c8-bbe5-98be13c329b4 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.600701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6365e89-f097-49c8-bbe5-98be13c329b4 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.600970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6365e89-f097-49c8-bbe5-98be13c329b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.622939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a6365e89-f097-49c8-bbe5-98be13c329b4 None None] MFA Rules not processed for user `8bfae94e0fa041c9802729370a810245`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:54:52.624151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a6365e89-f097-49c8-bbe5-98be13c329b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:52.630630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1013/1989] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:54:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:54:52.645130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7f7354-86b9-4e72-87bb-6b9a6b4fd56f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:54:52.646942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9d7f7354-86b9-4e72-87bb-6b9a6b4fd56f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:52.660641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7f7354-86b9-4e72-87bb-6b9a6b4fd56f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:54:52.664336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8782ed50-6a2b-4075-8b6d-a912234fc69d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:54:52.664994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8782ed50-6a2b-4075-8b6d-a912234fc69d service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.665276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8782ed50-6a2b-4075-8b6d-a912234fc69d service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.665549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8782ed50-6a2b-4075-8b6d-a912234fc69d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.672985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8782ed50-6a2b-4075-8b6d-a912234fc69d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:54:52.674409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8782ed50-6a2b-4075-8b6d-a912234fc69d service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:54:52.682704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 977/1990] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:54:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:54:52.770180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf53067-129b-4a0d-b0f1-5230a6104331 None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.770551 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf53067-129b-4a0d-b0f1-5230a6104331 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.770735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf53067-129b-4a0d-b0f1-5230a6104331 None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.772168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1014/1991] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:54:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:54:52.777075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91eb182c-8f00-4656-adad-b68198372fbd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.777362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91eb182c-8f00-4656-adad-b68198372fbd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.777635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91eb182c-8f00-4656-adad-b68198372fbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.797780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-91eb182c-8f00-4656-adad-b68198372fbd None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:54:52.798978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-91eb182c-8f00-4656-adad-b68198372fbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:52.809252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 978/1992] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:54:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:54:52.820060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf53067-129b-4a0d-b0f1-5230a6104331 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:54:52.830941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf53067-129b-4a0d-b0f1-5230a6104331 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:54:52.834551 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc8f3e1-07f2-4a80-ad1e-33666d4618e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:54:52.835143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8f3e1-07f2-4a80-ad1e-33666d4618e5 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:52.835421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8f3e1-07f2-4a80-ad1e-33666d4618e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:52.835746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8f3e1-07f2-4a80-ad1e-33666d4618e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:52.845897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc8f3e1-07f2-4a80-ad1e-33666d4618e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:54:52.847242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc8f3e1-07f2-4a80-ad1e-33666d4618e5 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:54:52.858775 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1015/1993] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:54:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:54:57.858321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b995b8-cd2c-4e59-996a-074dbce518d8 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:57.858812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b995b8-cd2c-4e59-996a-074dbce518d8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:57.858926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b995b8-cd2c-4e59-996a-074dbce518d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:57.879086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-22b995b8-cd2c-4e59-996a-074dbce518d8 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:54:57.880333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-22b995b8-cd2c-4e59-996a-074dbce518d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:54:57.887791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 979/1994] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:54:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:54:57.900474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc8f3e1-07f2-4a80-ad1e-33666d4618e5 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:54:57.911851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc8f3e1-07f2-4a80-ad1e-33666d4618e5 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:54:57.917790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aabc4b-c7d3-4d41-9120-133a2edddf8c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:54:57.918385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aabc4b-c7d3-4d41-9120-133a2edddf8c service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:54:57.918672 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aabc4b-c7d3-4d41-9120-133a2edddf8c service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:54:57.918947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aabc4b-c7d3-4d41-9120-133a2edddf8c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:54:57.927219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aabc4b-c7d3-4d41-9120-133a2edddf8c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:54:57.928828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aabc4b-c7d3-4d41-9120-133a2edddf8c service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:54:57.938393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1016/1995] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:54:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:55:07.874912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=93 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:55:07.934839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=41 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:55:25.360252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148bd425-0a47-4969-b3bc-686fb0f12bb2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:55:25.360991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148bd425-0a47-4969-b3bc-686fb0f12bb2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:55:25.360991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148bd425-0a47-4969-b3bc-686fb0f12bb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:55:25.381296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-148bd425-0a47-4969-b3bc-686fb0f12bb2 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:55:25.382631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-148bd425-0a47-4969-b3bc-686fb0f12bb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:55:25.389918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 980/1996] 10.222.0.39 () {58 vars in 967 bytes} [Mon Oct 24 08:55:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:55:25.402986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aabc4b-c7d3-4d41-9120-133a2edddf8c service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:55:25.414133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aabc4b-c7d3-4d41-9120-133a2edddf8c service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:55:25.419301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc97d998-cfb2-4b0f-97fe-63a1ad6026a9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:55:25.419904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc97d998-cfb2-4b0f-97fe-63a1ad6026a9 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:55:25.420218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc97d998-cfb2-4b0f-97fe-63a1ad6026a9 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:55:25.420499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc97d998-cfb2-4b0f-97fe-63a1ad6026a9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:55:25.429301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc97d998-cfb2-4b0f-97fe-63a1ad6026a9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:55:25.430877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc97d998-cfb2-4b0f-97fe-63a1ad6026a9 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:55:25.439007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1017/1997] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:55:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:55:35.375795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:55:35.435703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=6 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:55:51.465140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148bd425-0a47-4969-b3bc-686fb0f12bb2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:55:51.474611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148bd425-0a47-4969-b3bc-686fb0f12bb2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:55:51.478888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:55:51.479480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:55:51.479757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:55:51.480027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:55:51.481775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:55:51.495947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:55:51.497388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:55:51.509143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 981/1998] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:55:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:55:55.040966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc97d998-cfb2-4b0f-97fe-63a1ad6026a9 service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:55:55.050133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc97d998-cfb2-4b0f-97fe-63a1ad6026a9 service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:55:55.053700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b422ba3-7621-402e-bc21-00e4c5f49596 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:55:55.054283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b422ba3-7621-402e-bc21-00e4c5f49596 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:55:55.054560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b422ba3-7621-402e-bc21-00e4c5f49596 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:55:55.054828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b422ba3-7621-402e-bc21-00e4c5f49596 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:55:55.056582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4b422ba3-7621-402e-bc21-00e4c5f49596 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:55:55.070091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b422ba3-7621-402e-bc21-00e4c5f49596 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:55:55.071507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b422ba3-7621-402e-bc21-00e4c5f49596 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:55:55.082359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1018/1999] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:55:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:55:55.146932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:55:55.155515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618b5165-522c-4f05-82a9-c0b2ce2afe9d service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:55:55.159289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:55:55.159866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:55:55.160181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:55:55.160655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:55:55.163457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:55:55.222789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:55:55.224239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:55:55.233847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 982/2000] 10.222.0.39 () {60 vars in 1396 bytes} [Mon Oct 24 08:55:55 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3268 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:56:00.028409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54633f-3659-4d01-9587-1bdbf8f4079b None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:00.028954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54633f-3659-4d01-9587-1bdbf8f4079b None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:00.028954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54633f-3659-4d01-9587-1bdbf8f4079b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:00.048003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-7a54633f-3659-4d01-9587-1bdbf8f4079b None None] MFA Rules not processed for user `7127613bde784330bec364f02180291e`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:56:00.049235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7a54633f-3659-4d01-9587-1bdbf8f4079b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:00.057165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1019/2001] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:56:00 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:56:05.229463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=34 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:56:06.250869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:06.259986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26827caf-9a69-4c77-a674-c55f5617d867 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:06.263580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:06.264202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:06.264479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:06.264747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:06.266578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:06.277401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:06.278817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:06.287774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 983/2002] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:56:10.043049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=9 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:56:16.283697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=6 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:56:25.421942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a54633f-3659-4d01-9587-1bdbf8f4079b None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:25.423511 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7a54633f-3659-4d01-9587-1bdbf8f4079b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:25.439718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a54633f-3659-4d01-9587-1bdbf8f4079b None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:25.445949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:25.446556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:25.446838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:25.447112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:25.448812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:25.459431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:25.461125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:25.470513 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1020/2003] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:25 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:56:28.766994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:28.783718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491db8e-6f8c-43a9-ab4c-59e76b1eda7a service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:28.790589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7870d3bc-73cd-4e3a-8efb-440b9aa9632c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:28.791260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870d3bc-73cd-4e3a-8efb-440b9aa9632c service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:28.791546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870d3bc-73cd-4e3a-8efb-440b9aa9632c service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:28.791966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870d3bc-73cd-4e3a-8efb-440b9aa9632c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:28.793894 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7870d3bc-73cd-4e3a-8efb-440b9aa9632c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:28.816593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7870d3bc-73cd-4e3a-8efb-440b9aa9632c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:28.819641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7870d3bc-73cd-4e3a-8efb-440b9aa9632c service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:28.831799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 984/2004] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:56:34.758087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:34.769167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dfb9f9-dfd1-4c75-a1fa-1668a945b8d6 service nova] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:34.775363 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:34.775963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:34.776274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:34.776548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:34.778290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:34.835987 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:34.837708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:34.846628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1021/2005] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:34 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:56:38.826149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=6 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:56:42.260870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f3d7d1-cbfe-478e-9d35-849dfcd548a5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:42.261511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f3d7d1-cbfe-478e-9d35-849dfcd548a5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:42.261511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f3d7d1-cbfe-478e-9d35-849dfcd548a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:42.280610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-90f3d7d1-cbfe-478e-9d35-849dfcd548a5 None None] MFA Rules not processed for user `ebdef1d69bb14837b9a0c28cbba332fc`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:56:42.281896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-90f3d7d1-cbfe-478e-9d35-849dfcd548a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:42.290100 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 985/2006] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:56:42 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:56:42.606035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:42.616056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0696707b-65e9-4f9f-9482-777326513461 service nova] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:42.619842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46732ed4-3391-43d6-bf79-c016b9dbf845 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:42.620476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46732ed4-3391-43d6-bf79-c016b9dbf845 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:42.620759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46732ed4-3391-43d6-bf79-c016b9dbf845 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:42.621034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46732ed4-3391-43d6-bf79-c016b9dbf845 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:42.622739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-46732ed4-3391-43d6-bf79-c016b9dbf845 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:42.633231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46732ed4-3391-43d6-bf79-c016b9dbf845 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:42.634721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46732ed4-3391-43d6-bf79-c016b9dbf845 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:42.642709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1022/2007] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:56:42.984749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f3d7d1-cbfe-478e-9d35-849dfcd548a5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:42.986284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-90f3d7d1-cbfe-478e-9d35-849dfcd548a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:43.004327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f3d7d1-cbfe-478e-9d35-849dfcd548a5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:43.009670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:43.010293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:43.010573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:43.010846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:43.012634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:43.024944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:43.026546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:43.037330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 986/2008] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:56:45.182944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2ad99-5d51-4a36-8cfa-532bf5bbe32d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:45.183486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2ad99-5d51-4a36-8cfa-532bf5bbe32d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:45.183602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2ad99-5d51-4a36-8cfa-532bf5bbe32d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:45.207332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-74c2ad99-5d51-4a36-8cfa-532bf5bbe32d None None] MFA Rules not processed for user `552c23b8456a40ec8e01297aca7965d3`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:56:45.208849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-74c2ad99-5d51-4a36-8cfa-532bf5bbe32d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:45.221932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1023/2009] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:56:45 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:56:45.592865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:45.603358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effc0f42-6379-4a14-abfb-2f570443bc4e service placement] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:45.607875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:45.608641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:45.609154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:45.609504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:45.611326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:45.622465 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:45.624228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:45.634742 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 987/2010] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:56:51.116833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c2ad99-5d51-4a36-8cfa-532bf5bbe32d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:51.128370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=44 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:56:51.162926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c2ad99-5d51-4a36-8cfa-532bf5bbe32d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:51.171768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5eed04f-dd2b-4f14-b964-8bb91ca23f5e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:51.172709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eed04f-dd2b-4f14-b964-8bb91ca23f5e service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:51.173187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eed04f-dd2b-4f14-b964-8bb91ca23f5e service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:51.173652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eed04f-dd2b-4f14-b964-8bb91ca23f5e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:51.177884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b5eed04f-dd2b-4f14-b964-8bb91ca23f5e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:51.196413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5eed04f-dd2b-4f14-b964-8bb91ca23f5e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:51.204986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5eed04f-dd2b-4f14-b964-8bb91ca23f5e service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:51.218389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1024/2011] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 08:56:51.453686 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:56:51.469456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fdc630-c40e-44ee-8475-69d6017daede service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:56:51.476439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca8bb75-baea-4cdc-9cac-81c855856eb8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:56:51.477052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca8bb75-baea-4cdc-9cac-81c855856eb8 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:56:51.477331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca8bb75-baea-4cdc-9cac-81c855856eb8 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:56:51.477794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca8bb75-baea-4cdc-9cac-81c855856eb8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:56:51.479547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-0ca8bb75-baea-4cdc-9cac-81c855856eb8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:56:51.493414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca8bb75-baea-4cdc-9cac-81c855856eb8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:56:51.495113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca8bb75-baea-4cdc-9cac-81c855856eb8 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:56:51.507225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 988/2012] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:56:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:01.214228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:01.501532 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=21 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:17.333921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414a8e4-051a-4b98-925b-f4fa0769c71f None None] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:17.334534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414a8e4-051a-4b98-925b-f4fa0769c71f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:17.334534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414a8e4-051a-4b98-925b-f4fa0769c71f None None] PATH_INFO: `/` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:17.336094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1025/2013] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:57:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:57:17.342594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc27124-6794-4024-a55e-7c41a86d39fa None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:17.343262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc27124-6794-4024-a55e-7c41a86d39fa None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:17.343262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc27124-6794-4024-a55e-7c41a86d39fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:17.364377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-bcc27124-6794-4024-a55e-7c41a86d39fa None None] MFA Rules not processed for user `8bfae94e0fa041c9802729370a810245`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:17.365616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bcc27124-6794-4024-a55e-7c41a86d39fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:17.372198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 989/2014] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:57:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:17.385566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3414a8e4-051a-4b98-925b-f4fa0769c71f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:17.396958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3414a8e4-051a-4b98-925b-f4fa0769c71f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:17.400929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aaa8485-20ab-4d73-8bae-87840b9fb712 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:17.401544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaa8485-20ab-4d73-8bae-87840b9fb712 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:17.401821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaa8485-20ab-4d73-8bae-87840b9fb712 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:17.402103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaa8485-20ab-4d73-8bae-87840b9fb712 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:17.410842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aaa8485-20ab-4d73-8bae-87840b9fb712 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:17.412316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aaa8485-20ab-4d73-8bae-87840b9fb712 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:17.421101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1026/2015] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:17.513202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fdaaf8-cfd6-4971-9b30-134e8cc655d2 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:17.513480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fdaaf8-cfd6-4971-9b30-134e8cc655d2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:17.513751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fdaaf8-cfd6-4971-9b30-134e8cc655d2 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:17.515105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 990/2016] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:57:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:57:17.521614 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5833fecb-5ca8-43f7-b2b6-86ec702c880a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:17.522136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5833fecb-5ca8-43f7-b2b6-86ec702c880a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:17.522418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5833fecb-5ca8-43f7-b2b6-86ec702c880a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:17.544000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5833fecb-5ca8-43f7-b2b6-86ec702c880a None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:17.545374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5833fecb-5ca8-43f7-b2b6-86ec702c880a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:17.556426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1027/2017] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:57:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:17.569720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fdaaf8-cfd6-4971-9b30-134e8cc655d2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:17.596309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fdaaf8-cfd6-4971-9b30-134e8cc655d2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:17.602614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:17.603554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:17.604007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:17.604489 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:17.622135 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:17.624835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:17.641767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 991/2018] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:22.314435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5833fecb-5ca8-43f7-b2b6-86ec702c880a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:22.324003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5833fecb-5ca8-43f7-b2b6-86ec702c880a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:22.327617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:22.328267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:22.328553 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:22.328824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:22.332064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:22.413962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:22.415327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:22.427622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1028/2019] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:26.265687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:26.267154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:26.283374 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8a8d97-6bb3-43e3-9649-c0f2cabc494c service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:26.289756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:26.290511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:26.290791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:26.291060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] PATH_INFO: `/v3/users/7127613bde784330bec364f02180291e` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:26.292211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7127613bde784330bec364f02180291e)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:26.294150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:26.347367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7127613bde784330bec364f02180291e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.354875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7127613bde784330bec364f02180291e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.367971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7127613bde784330bec364f02180291e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.368761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7127613bde784330bec364f02180291e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.369375 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f5275ffa-4523-4f4d-be6d-650a62f715b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7127613bde784330bec364f02180291e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.383418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 992/2020] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:57:26 2022] DELETE /identity/v3/users/7127613bde784330bec364f02180291e => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:57:26.559874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:26.570721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca6cf12-ffb5-4db7-8322-3a7996abc28b service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:26.631261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:26.631898 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:26.632214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:26.632493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] PATH_INFO: `/v3/projects/fd4b77e78f9d4286aa92f6f90ff8b797` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:26.633611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd4b77e78f9d4286aa92f6f90ff8b797)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:26.635512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:26.685393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd4b77e78f9d4286aa92f6f90ff8b797', 'request_id': 'req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.692786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd4b77e78f9d4286aa92f6f90ff8b797', 'request_id': 'req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.700737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd4b77e78f9d4286aa92f6f90ff8b797', 'request_id': 'req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.712728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] The token cache is being invalidate because project fd4b77e78f9d4286aa92f6f90ff8b797 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:57:26.713017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.713665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fa0a9987-cd6a-4898-aa0e-36e7a00aa84a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:26.715154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1029/2021] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:57:26 2022] DELETE /identity/v3/projects/fd4b77e78f9d4286aa92f6f90ff8b797 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:26.727720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8333731f-2064-4003-9d99-48da99d24dff None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:26.728011 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8333731f-2064-4003-9d99-48da99d24dff None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:26.728304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8333731f-2064-4003-9d99-48da99d24dff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:26.753885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-8333731f-2064-4003-9d99-48da99d24dff None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:26.755131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-8333731f-2064-4003-9d99-48da99d24dff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:26.824476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 993/2022] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:26.833305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acb3f97-91b0-4924-b357-458990a11118 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:26.833633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acb3f97-91b0-4924-b357-458990a11118 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:26.833891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acb3f97-91b0-4924-b357-458990a11118 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:26.842675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:26.856518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0acb3f97-91b0-4924-b357-458990a11118 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:26.857722 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0acb3f97-91b0-4924-b357-458990a11118 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:26.866956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1030/2023] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:57:26.871302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8333731f-2064-4003-9d99-48da99d24dff None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:26.882857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8333731f-2064-4003-9d99-48da99d24dff None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:26.887455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:26.888056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:26.888362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:26.888633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:26.889272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:57:26.889568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:26.891155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:26.903886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 994/2024] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:57:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:57:26.908346 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acb3f97-91b0-4924-b357-458990a11118 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:26.918022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acb3f97-91b0-4924-b357-458990a11118 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:26.923029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af680bf-713b-45d2-ab5a-b10883e70627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:26.923650 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af680bf-713b-45d2-ab5a-b10883e70627 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:26.923933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af680bf-713b-45d2-ab5a-b10883e70627 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:26.925227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af680bf-713b-45d2-ab5a-b10883e70627 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:26.926253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af680bf-713b-45d2-ab5a-b10883e70627 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:26.928358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af680bf-713b-45d2-ab5a-b10883e70627 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:26.961017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1031/2025] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:57:26 2022] POST /identity/v3/projects => generated 391 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:57:26.966341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:26.984841 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ddf066-3c8d-4f5d-8b71-a4776b86bb3f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.035914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:27.057311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af683c6-0b3f-41b8-92ae-1ce643369158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.057999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af683c6-0b3f-41b8-92ae-1ce643369158 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.058280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af683c6-0b3f-41b8-92ae-1ce643369158 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.058554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af683c6-0b3f-41b8-92ae-1ce643369158 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.059206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af683c6-0b3f-41b8-92ae-1ce643369158 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.061208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af683c6-0b3f-41b8-92ae-1ce643369158 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.091905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 995/2026] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:57:26 2022] POST /identity/v3/users => generated 296 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:57:27.096213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af680bf-713b-45d2-ab5a-b10883e70627 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.107517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af680bf-713b-45d2-ab5a-b10883e70627 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.113422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91ac4bd-0660-4d4f-8bd9-e86f36888633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.114500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91ac4bd-0660-4d4f-8bd9-e86f36888633 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.114788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91ac4bd-0660-4d4f-8bd9-e86f36888633 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.115066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91ac4bd-0660-4d4f-8bd9-e86f36888633 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.115711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91ac4bd-0660-4d4f-8bd9-e86f36888633 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.117398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91ac4bd-0660-4d4f-8bd9-e86f36888633 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.128679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1032/2027] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:57:27 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:57:27.133381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af683c6-0b3f-41b8-92ae-1ce643369158 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.144046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af683c6-0b3f-41b8-92ae-1ce643369158 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.148570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeab8a5-68bd-4fff-a1d5-966a68e1dd5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.149398 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeab8a5-68bd-4fff-a1d5-966a68e1dd5e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.149550 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeab8a5-68bd-4fff-a1d5-966a68e1dd5e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.149735 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeab8a5-68bd-4fff-a1d5-966a68e1dd5e admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.150390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abeab8a5-68bd-4fff-a1d5-966a68e1dd5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.151999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abeab8a5-68bd-4fff-a1d5-966a68e1dd5e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.166021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 996/2028] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:57:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:57:27.170530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91ac4bd-0660-4d4f-8bd9-e86f36888633 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.181529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91ac4bd-0660-4d4f-8bd9-e86f36888633 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.186511 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bfb93c-643a-45c5-abbb-941457d56d6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.187179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bfb93c-643a-45c5-abbb-941457d56d6f admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.187457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bfb93c-643a-45c5-abbb-941457d56d6f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.187731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bfb93c-643a-45c5-abbb-941457d56d6f admin admin] PATH_INFO: `/v3/projects/f34632fdf67642cc9f176a719aaf8b26/users/57f028d175044334888ed30bc3472d8d/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.206713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bfb93c-643a-45c5-abbb-941457d56d6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f34632fdf67642cc9f176a719aaf8b26, user_id=57f028d175044334888ed30bc3472d8d, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.209576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bfb93c-643a-45c5-abbb-941457d56d6f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.225962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1033/2029] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:57:27 2022] PUT /identity/v3/projects/f34632fdf67642cc9f176a719aaf8b26/users/57f028d175044334888ed30bc3472d8d/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:27.238521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08171921-05fa-43c6-b113-3450f1e2d0e8 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.239165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08171921-05fa-43c6-b113-3450f1e2d0e8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.239395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08171921-05fa-43c6-b113-3450f1e2d0e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.277426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-08171921-05fa-43c6-b113-3450f1e2d0e8 None None] MFA Rules not processed for user `57f028d175044334888ed30bc3472d8d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:27.279414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-08171921-05fa-43c6-b113-3450f1e2d0e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:27.362954 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 997/2030] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:27 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:27.367765 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bfb93c-643a-45c5-abbb-941457d56d6f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.435968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bfb93c-643a-45c5-abbb-941457d56d6f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.442191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3078233d-ccbb-4c4f-808d-7a74af3dd9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.442792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3078233d-ccbb-4c4f-808d-7a74af3dd9ac admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.443066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3078233d-ccbb-4c4f-808d-7a74af3dd9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.443332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3078233d-ccbb-4c4f-808d-7a74af3dd9ac admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.443971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3078233d-ccbb-4c4f-808d-7a74af3dd9ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.445912 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3078233d-ccbb-4c4f-808d-7a74af3dd9ac admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.477044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1034/2031] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:57:27 2022] POST /identity/v3/projects => generated 393 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:57:27.481084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08171921-05fa-43c6-b113-3450f1e2d0e8 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.494324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08171921-05fa-43c6-b113-3450f1e2d0e8 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.554252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab6d6bd-e740-49b1-a919-b23f0aadf7c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.554864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab6d6bd-e740-49b1-a919-b23f0aadf7c2 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.555146 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab6d6bd-e740-49b1-a919-b23f0aadf7c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.555421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab6d6bd-e740-49b1-a919-b23f0aadf7c2 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.556073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab6d6bd-e740-49b1-a919-b23f0aadf7c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.558043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab6d6bd-e740-49b1-a919-b23f0aadf7c2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.583621 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 998/2032] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:57:27 2022] POST /identity/v3/users => generated 303 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:57:27.587851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3078233d-ccbb-4c4f-808d-7a74af3dd9ac admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.599685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3078233d-ccbb-4c4f-808d-7a74af3dd9ac admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.605006 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad07ee-cbc9-4948-bdf0-4d5fbe170bfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.605751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad07ee-cbc9-4948-bdf0-4d5fbe170bfd admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.606078 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad07ee-cbc9-4948-bdf0-4d5fbe170bfd admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.606413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad07ee-cbc9-4948-bdf0-4d5fbe170bfd admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.607113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ad07ee-cbc9-4948-bdf0-4d5fbe170bfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.608946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ad07ee-cbc9-4948-bdf0-4d5fbe170bfd admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.622120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1035/2033] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:57:27 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:57:27.627173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab6d6bd-e740-49b1-a919-b23f0aadf7c2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.638654 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab6d6bd-e740-49b1-a919-b23f0aadf7c2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.644161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71225f-523d-4450-a5b4-8151f3aa6492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.644826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71225f-523d-4450-a5b4-8151f3aa6492 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.645203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71225f-523d-4450-a5b4-8151f3aa6492 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.645398 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71225f-523d-4450-a5b4-8151f3aa6492 admin admin] PATH_INFO: `/v3/projects/d1d78077c9944815be64e9879b264e17/users/0484bc1f232f420c9692efef9637fe1e/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.660774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71225f-523d-4450-a5b4-8151f3aa6492 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1d78077c9944815be64e9879b264e17, user_id=0484bc1f232f420c9692efef9637fe1e, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.663433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71225f-523d-4450-a5b4-8151f3aa6492 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.678304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 999/2034] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:57:27 2022] PUT /identity/v3/projects/d1d78077c9944815be64e9879b264e17/users/0484bc1f232f420c9692efef9637fe1e/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:57:27.682977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad07ee-cbc9-4948-bdf0-4d5fbe170bfd admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.695001 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad07ee-cbc9-4948-bdf0-4d5fbe170bfd admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.761805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014ba2f2-d99a-4fd9-8744-7966b0308c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.762409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ba2f2-d99a-4fd9-8744-7966b0308c7f admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.762686 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ba2f2-d99a-4fd9-8744-7966b0308c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.762953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ba2f2-d99a-4fd9-8744-7966b0308c7f admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.763587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014ba2f2-d99a-4fd9-8744-7966b0308c7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.765196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014ba2f2-d99a-4fd9-8744-7966b0308c7f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.775491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1036/2035] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:57:27 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:57:27.779884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71225f-523d-4450-a5b4-8151f3aa6492 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.790078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71225f-523d-4450-a5b4-8151f3aa6492 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:27.794880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:27.795529 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.795807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.796079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] PATH_INFO: `/v3/projects/d1d78077c9944815be64e9879b264e17/users/0484bc1f232f420c9692efef9637fe1e/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.797947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1d78077c9944815be64e9879b264e17, user_id=0484bc1f232f420c9692efef9637fe1e, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:27.801771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:27.808696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:27.814679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1000/2036] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:57:27 2022] PUT /identity/v3/projects/d1d78077c9944815be64e9879b264e17/users/0484bc1f232f420c9692efef9637fe1e/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:27.826123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffc4a75-9713-4a36-824b-9db297576e23 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:27.826404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffc4a75-9713-4a36-824b-9db297576e23 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:27.826674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffc4a75-9713-4a36-824b-9db297576e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:27.847768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:27.862618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-3ffc4a75-9713-4a36-824b-9db297576e23 None None] MFA Rules not processed for user `0484bc1f232f420c9692efef9637fe1e`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:27.864919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3ffc4a75-9713-4a36-824b-9db297576e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:27.944090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1037/2037] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:27 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:57:27.957268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:27.959950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:28.045582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed0273-29bb-4e9d-a0dd-b1e5824f89ac admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:28.050933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3087fe57-df3a-42d7-bd02-cd97c48d71e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:28.051548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3087fe57-df3a-42d7-bd02-cd97c48d71e3 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:28.051823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3087fe57-df3a-42d7-bd02-cd97c48d71e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:28.052098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3087fe57-df3a-42d7-bd02-cd97c48d71e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:28.061944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3087fe57-df3a-42d7-bd02-cd97c48d71e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:28.063391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3087fe57-df3a-42d7-bd02-cd97c48d71e3 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:28.074522 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1001/2038] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:27 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:28.125209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffc4a75-9713-4a36-824b-9db297576e23 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:28.126655 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3ffc4a75-9713-4a36-824b-9db297576e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:28.139732 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffc4a75-9713-4a36-824b-9db297576e23 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:28.143781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93465fd8-13c9-4da1-bd12-057688e63c7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:28.144667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93465fd8-13c9-4da1-bd12-057688e63c7e service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:28.144949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93465fd8-13c9-4da1-bd12-057688e63c7e service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:28.145228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93465fd8-13c9-4da1-bd12-057688e63c7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:28.201956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93465fd8-13c9-4da1-bd12-057688e63c7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:28.203285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93465fd8-13c9-4da1-bd12-057688e63c7e service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:28.212514 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1038/2039] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:28 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:31.588709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3087fe57-df3a-42d7-bd02-cd97c48d71e3 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:31.655943 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3087fe57-df3a-42d7-bd02-cd97c48d71e3 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:31.661214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:31.661844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:31.662122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:31.662393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] PATH_INFO: `/v3/users/57f028d175044334888ed30bc3472d8d` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:31.663513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57f028d175044334888ed30bc3472d8d)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:31.665483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:31.716812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57f028d175044334888ed30bc3472d8d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:31.724793 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57f028d175044334888ed30bc3472d8d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:31.737006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57f028d175044334888ed30bc3472d8d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:31.738052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57f028d175044334888ed30bc3472d8d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:31.738699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57f028d175044334888ed30bc3472d8d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:31.753786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1002/2040] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:57:31 2022] DELETE /identity/v3/users/57f028d175044334888ed30bc3472d8d => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:31.758360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3bfd9-9966-494f-a528-241daa0cc92e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:31.758905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3bfd9-9966-494f-a528-241daa0cc92e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:31.758905 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3bfd9-9966-494f-a528-241daa0cc92e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:31.778983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5dc3bfd9-9966-494f-a528-241daa0cc92e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:31.780239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5dc3bfd9-9966-494f-a528-241daa0cc92e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:31.840805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1039/2041] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:31.850644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:31.852071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:31.885707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:31.948015 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168191bb-f8f7-47f6-a250-4b6196e7f2d4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:31.955796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c30cc1-ffb5-4d91-b961-0416b0d613b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:31.957914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c30cc1-ffb5-4d91-b961-0416b0d613b7 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:31.957914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c30cc1-ffb5-4d91-b961-0416b0d613b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:31.958371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c30cc1-ffb5-4d91-b961-0416b0d613b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:31.976056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c30cc1-ffb5-4d91-b961-0416b0d613b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:31.979580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c30cc1-ffb5-4d91-b961-0416b0d613b7 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:31.998376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1003/2042] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:32.333310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc3bfd9-9966-494f-a528-241daa0cc92e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:32.335438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5dc3bfd9-9966-494f-a528-241daa0cc92e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:32.350055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc3bfd9-9966-494f-a528-241daa0cc92e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:32.355423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:32.356125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:32.356449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:32.356721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] PATH_INFO: `/v3/projects/f34632fdf67642cc9f176a719aaf8b26` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:32.358023 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f34632fdf67642cc9f176a719aaf8b26)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:32.360055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:32.415740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:32.438274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f34632fdf67642cc9f176a719aaf8b26', 'request_id': 'req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.447137 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f34632fdf67642cc9f176a719aaf8b26', 'request_id': 'req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.455001 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f34632fdf67642cc9f176a719aaf8b26', 'request_id': 'req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.467471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] The token cache is being invalidate because project f34632fdf67642cc9f176a719aaf8b26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:57:32.467760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.468456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.469989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1040/2043] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:57:32 2022] DELETE /identity/v3/projects/f34632fdf67642cc9f176a719aaf8b26 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:32.474586 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c30cc1-ffb5-4d91-b961-0416b0d613b7 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:32.484486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c30cc1-ffb5-4d91-b961-0416b0d613b7 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:32.546698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:32.547333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:32.547615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:32.547898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] PATH_INFO: `/v3/users/0484bc1f232f420c9692efef9637fe1e` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:32.549112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0484bc1f232f420c9692efef9637fe1e)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:32.551754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:32.609469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0484bc1f232f420c9692efef9637fe1e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.618794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0484bc1f232f420c9692efef9637fe1e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.633690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0484bc1f232f420c9692efef9637fe1e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.634451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0484bc1f232f420c9692efef9637fe1e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.635061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-bc3a2743-fd36-4486-b44f-3b7657b6f80c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0484bc1f232f420c9692efef9637fe1e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:32.650161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1004/2044] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:57:32 2022] DELETE /identity/v3/users/0484bc1f232f420c9692efef9637fe1e => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:33.005368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:33.007372 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:33.029786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fa7ccf-02c9-41d5-8862-4586e1aaf76d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:33.100889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:33.101540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.101819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.102092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] PATH_INFO: `/v3/projects/d1d78077c9944815be64e9879b264e17` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.103310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1d78077c9944815be64e9879b264e17)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:33.105447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:33.165070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1d78077c9944815be64e9879b264e17', 'request_id': 'req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:33.171920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1d78077c9944815be64e9879b264e17', 'request_id': 'req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:33.181907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1d78077c9944815be64e9879b264e17', 'request_id': 'req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:33.210187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] The token cache is being invalidate because project d1d78077c9944815be64e9879b264e17 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:57:33.210741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:33.216507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-a1748050-09fa-49bf-8bd1-2704ad5dfb4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:33.219306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1041/2045] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:57:33 2022] DELETE /identity/v3/projects/d1d78077c9944815be64e9879b264e17 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:57:33.225913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057207e-6dfe-4a25-b632-de4874d13480 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.226314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057207e-6dfe-4a25-b632-de4874d13480 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.226691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057207e-6dfe-4a25-b632-de4874d13480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.248619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a057207e-6dfe-4a25-b632-de4874d13480 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:33.250028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a057207e-6dfe-4a25-b632-de4874d13480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:33.302283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:33.339913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1005/2046] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:33.343600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b371fcbe-fd41-4059-9824-d62d8dbd7b1d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.343909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b371fcbe-fd41-4059-9824-d62d8dbd7b1d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.344214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b371fcbe-fd41-4059-9824-d62d8dbd7b1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.368388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-b371fcbe-fd41-4059-9824-d62d8dbd7b1d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:33.369666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b371fcbe-fd41-4059-9824-d62d8dbd7b1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:33.378625 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1042/2047] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:33.383852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a057207e-6dfe-4a25-b632-de4874d13480 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:33.395294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a057207e-6dfe-4a25-b632-de4874d13480 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:33.400599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:33.401291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.401594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.401844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.402748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:57:33.403060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:33.404896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:33.426614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1006/2048] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:57:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:57:33.427821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b371fcbe-fd41-4059-9824-d62d8dbd7b1d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:33.446696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b371fcbe-fd41-4059-9824-d62d8dbd7b1d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:33.452274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebadd28-e8c4-4725-9245-b492d9b65484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:33.452891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebadd28-e8c4-4725-9245-b492d9b65484 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.453200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebadd28-e8c4-4725-9245-b492d9b65484 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.453469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebadd28-e8c4-4725-9245-b492d9b65484 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.454298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebadd28-e8c4-4725-9245-b492d9b65484 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:33.456828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebadd28-e8c4-4725-9245-b492d9b65484 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:33.489543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1043/2049] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:57:33 2022] POST /identity/v3/projects => generated 375 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:57:33.494752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:33.505863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1864642-b8f2-428b-a3bd-ce4d32c4a705 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:33.566934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f729f2-51f8-4f23-a229-0f0aed864f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:33.567557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f729f2-51f8-4f23-a229-0f0aed864f06 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.567840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f729f2-51f8-4f23-a229-0f0aed864f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.568114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f729f2-51f8-4f23-a229-0f0aed864f06 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.568793 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f729f2-51f8-4f23-a229-0f0aed864f06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:33.570739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f729f2-51f8-4f23-a229-0f0aed864f06 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:33.598042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1007/2050] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:57:33 2022] POST /identity/v3/users => generated 288 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:57:33.602267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebadd28-e8c4-4725-9245-b492d9b65484 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:33.614718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebadd28-e8c4-4725-9245-b492d9b65484 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:33.619724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c586fb79-e28e-4efc-ac33-1d61af40cd30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:33.620383 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c586fb79-e28e-4efc-ac33-1d61af40cd30 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.620666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c586fb79-e28e-4efc-ac33-1d61af40cd30 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.620939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c586fb79-e28e-4efc-ac33-1d61af40cd30 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.621625 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c586fb79-e28e-4efc-ac33-1d61af40cd30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:33.623255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c586fb79-e28e-4efc-ac33-1d61af40cd30 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:33.635678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1044/2051] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:57:33.640403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f729f2-51f8-4f23-a229-0f0aed864f06 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:33.652434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f729f2-51f8-4f23-a229-0f0aed864f06 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:33.657579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f1ba85-b7e7-40f1-8f0e-b47c4451704e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:33.658292 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f1ba85-b7e7-40f1-8f0e-b47c4451704e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.658579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f1ba85-b7e7-40f1-8f0e-b47c4451704e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.659582 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f1ba85-b7e7-40f1-8f0e-b47c4451704e admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.660543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f1ba85-b7e7-40f1-8f0e-b47c4451704e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:33.662196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f1ba85-b7e7-40f1-8f0e-b47c4451704e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:33.674846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1008/2052] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:57:33.681084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c586fb79-e28e-4efc-ac33-1d61af40cd30 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:33.694995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c586fb79-e28e-4efc-ac33-1d61af40cd30 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:33.699874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:33.700586 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.700868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.701191 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] PATH_INFO: `/v3/projects/f55bd1eceb4c42448ebbcd5e676c8310/users/5468d56912ef451881969b5d09a0a178/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.705200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:33.721208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f55bd1eceb4c42448ebbcd5e676c8310, user_id=5468d56912ef451881969b5d09a0a178, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:33.725233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:33.747853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1045/2053] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:57:33 2022] PUT /identity/v3/projects/f55bd1eceb4c42448ebbcd5e676c8310/users/5468d56912ef451881969b5d09a0a178/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:33.754997 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c338e8-b501-44ab-8eae-ce958e0163f0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.755310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c338e8-b501-44ab-8eae-ce958e0163f0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.755587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c338e8-b501-44ab-8eae-ce958e0163f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.788462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-45c338e8-b501-44ab-8eae-ce958e0163f0 None None] MFA Rules not processed for user `5468d56912ef451881969b5d09a0a178`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:33.789689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-45c338e8-b501-44ab-8eae-ce958e0163f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:33.861423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1009/2054] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:33 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:57:33.882163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:33.884310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:33.947727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cf5f72-76e0-4506-8687-7a69387b4939 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:33.951438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab09047-3c98-4726-ac07-7055c90c945b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a4fd11d5d85479c9aa1d87b23a13141', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:33.952041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab09047-3c98-4726-ac07-7055c90c945b service glance] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:33.952347 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab09047-3c98-4726-ac07-7055c90c945b service glance] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:33.952616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab09047-3c98-4726-ac07-7055c90c945b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:33.962696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab09047-3c98-4726-ac07-7055c90c945b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:33.964160 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab09047-3c98-4726-ac07-7055c90c945b service glance] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:33.975010 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1046/2055] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:33 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:43.857425 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=79,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:43.969470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=32,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:58.056924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c338e8-b501-44ab-8eae-ce958e0163f0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:58.121079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c338e8-b501-44ab-8eae-ce958e0163f0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:58.125619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:58.126245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:58.127110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:58.127435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] PATH_INFO: `/v3/users/5468d56912ef451881969b5d09a0a178` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:58.128627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5468d56912ef451881969b5d09a0a178)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:58.130545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:58.181576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5468d56912ef451881969b5d09a0a178'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.188537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5468d56912ef451881969b5d09a0a178'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.200929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5468d56912ef451881969b5d09a0a178'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.201734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5468d56912ef451881969b5d09a0a178'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.202232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5468d56912ef451881969b5d09a0a178'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.215759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1010/2056] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:57:58 2022] DELETE /identity/v3/users/5468d56912ef451881969b5d09a0a178 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:58.220266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3ca13-2605-43c7-9207-5e95bafa2701 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:58.224830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3ca13-2605-43c7-9207-5e95bafa2701 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:58.224830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3ca13-2605-43c7-9207-5e95bafa2701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:58.241401 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-22a3ca13-2605-43c7-9207-5e95bafa2701 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:58.242589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-22a3ca13-2605-43c7-9207-5e95bafa2701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:58.301583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1047/2057] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:58.311268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:58.312714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:58.371076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafb4e18-472e-4acd-bab3-be03ae208f44 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:58.374768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923da75-ae71-4938-9ec7-3d72765d19ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:58.375380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923da75-ae71-4938-9ec7-3d72765d19ac service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:58.375658 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923da75-ae71-4938-9ec7-3d72765d19ac service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:58.375929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923da75-ae71-4938-9ec7-3d72765d19ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:58.386115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d923da75-ae71-4938-9ec7-3d72765d19ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:58.387553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d923da75-ae71-4938-9ec7-3d72765d19ac service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:58.397662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1011/2058] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:57:58.671632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3ca13-2605-43c7-9207-5e95bafa2701 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:58.673186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-22a3ca13-2605-43c7-9207-5e95bafa2701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:58.685814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3ca13-2605-43c7-9207-5e95bafa2701 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:58.691235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:58.691938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:58.692316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:58.692705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] PATH_INFO: `/v3/projects/f55bd1eceb4c42448ebbcd5e676c8310` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:58.693853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f55bd1eceb4c42448ebbcd5e676c8310)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:58.695859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:58.744493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f55bd1eceb4c42448ebbcd5e676c8310', 'request_id': 'req-0fd37d3f-e755-48ae-8702-eb098d7519a1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.750919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f55bd1eceb4c42448ebbcd5e676c8310', 'request_id': 'req-0fd37d3f-e755-48ae-8702-eb098d7519a1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.759458 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f55bd1eceb4c42448ebbcd5e676c8310', 'request_id': 'req-0fd37d3f-e755-48ae-8702-eb098d7519a1'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.771749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] The token cache is being invalidate because project f55bd1eceb4c42448ebbcd5e676c8310 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:57:58.772038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.772710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0fd37d3f-e755-48ae-8702-eb098d7519a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:57:58.774181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1048/2059] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:57:58 2022] DELETE /identity/v3/projects/f55bd1eceb4c42448ebbcd5e676c8310 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:58.779110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fd1da-5d93-4333-a992-f67faf94ebe1 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:58.779396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fd1da-5d93-4333-a992-f67faf94ebe1 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:58.779667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fd1da-5d93-4333-a992-f67faf94ebe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:58.799667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-cc5fd1da-5d93-4333-a992-f67faf94ebe1 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:58.800826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cc5fd1da-5d93-4333-a992-f67faf94ebe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:58.817639 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:58.860938 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1012/2060] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:58.869050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bab13f7-36c0-4fb5-ba4e-ec129a3e9a4d None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:58.869331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bab13f7-36c0-4fb5-ba4e-ec129a3e9a4d None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:58.869601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bab13f7-36c0-4fb5-ba4e-ec129a3e9a4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:58.889798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-1bab13f7-36c0-4fb5-ba4e-ec129a3e9a4d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:58.890998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1bab13f7-36c0-4fb5-ba4e-ec129a3e9a4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:58.899351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1049/2061] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:58.903581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5fd1da-5d93-4333-a992-f67faf94ebe1 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:58.914037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5fd1da-5d93-4333-a992-f67faf94ebe1 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:58.918533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:58.919149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:58.919426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:58.919722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:58.920421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:57:58.920719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:58.922305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:58.935176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1013/2062] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:57:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:57:58.939536 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bab13f7-36c0-4fb5-ba4e-ec129a3e9a4d None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:58.948639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bab13f7-36c0-4fb5-ba4e-ec129a3e9a4d None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:58.953216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6818e0cd-061a-45ad-8b80-f084036871bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:58.953800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818e0cd-061a-45ad-8b80-f084036871bd admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:58.954075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818e0cd-061a-45ad-8b80-f084036871bd admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:58.954342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818e0cd-061a-45ad-8b80-f084036871bd admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:58.955085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6818e0cd-061a-45ad-8b80-f084036871bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:58.957014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6818e0cd-061a-45ad-8b80-f084036871bd admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:58.984950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1050/2063] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:57:58 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:57:58.990253 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:59.000803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f6c75-6bb1-4e51-8161-9b176acadc4e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:59.059714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10e6b1b-68f7-4ae0-910e-a11d02a24f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:59.060450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10e6b1b-68f7-4ae0-910e-a11d02a24f13 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:59.060559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10e6b1b-68f7-4ae0-910e-a11d02a24f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:59.060754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10e6b1b-68f7-4ae0-910e-a11d02a24f13 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:59.061382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10e6b1b-68f7-4ae0-910e-a11d02a24f13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:59.063271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10e6b1b-68f7-4ae0-910e-a11d02a24f13 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:59.089681 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1014/2064] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:57:58 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:57:59.090892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6818e0cd-061a-45ad-8b80-f084036871bd admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:59.102248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6818e0cd-061a-45ad-8b80-f084036871bd admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:59.106784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6019b12-5f72-4fde-86ba-ee72ad4b512b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:59.107366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6019b12-5f72-4fde-86ba-ee72ad4b512b admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:59.107642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6019b12-5f72-4fde-86ba-ee72ad4b512b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:59.107907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6019b12-5f72-4fde-86ba-ee72ad4b512b admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:59.108567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6019b12-5f72-4fde-86ba-ee72ad4b512b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:59.110132 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6019b12-5f72-4fde-86ba-ee72ad4b512b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:59.121144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1051/2065] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:57:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:57:59.125213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10e6b1b-68f7-4ae0-910e-a11d02a24f13 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:59.136177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10e6b1b-68f7-4ae0-910e-a11d02a24f13 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:59.140824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcab56df-7f90-4ada-8d61-302355e7b1bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:59.141413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab56df-7f90-4ada-8d61-302355e7b1bd admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:59.141696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab56df-7f90-4ada-8d61-302355e7b1bd admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:59.141970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab56df-7f90-4ada-8d61-302355e7b1bd admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:59.142820 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcab56df-7f90-4ada-8d61-302355e7b1bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:59.144443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcab56df-7f90-4ada-8d61-302355e7b1bd admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:59.155631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1015/2066] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:57:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 08:57:59.159917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6019b12-5f72-4fde-86ba-ee72ad4b512b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:59.169711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6019b12-5f72-4fde-86ba-ee72ad4b512b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:59.175241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:59.175875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:59.176177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:59.176452 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] PATH_INFO: `/v3/projects/c722bfda48424b8fb600dd3f8150e29b/users/0d7babaf2e3649d393478fa3ec9907ef/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:59.191833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c722bfda48424b8fb600dd3f8150e29b, user_id=0d7babaf2e3649d393478fa3ec9907ef, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:59.194506 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:59.208354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1052/2067] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:57:59 2022] PUT /identity/v3/projects/c722bfda48424b8fb600dd3f8150e29b/users/0d7babaf2e3649d393478fa3ec9907ef/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:57:59.219649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b839235-2e28-4db2-98eb-e7eb3ea06fc6 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:59.219940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b839235-2e28-4db2-98eb-e7eb3ea06fc6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:59.220260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b839235-2e28-4db2-98eb-e7eb3ea06fc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:59.249085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7b839235-2e28-4db2-98eb-e7eb3ea06fc6 None None] MFA Rules not processed for user `0d7babaf2e3649d393478fa3ec9907ef`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:57:59.250303 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7b839235-2e28-4db2-98eb-e7eb3ea06fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:59.313757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1016/2068] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:57:59 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:57:59.325040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:57:59.326796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:57:59.350583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:57:59.388232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49463e68-226f-4549-bd73-7b9214f47353 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:57:59.391981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b4e455-58eb-421d-9bc8-e784e806352d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:57:59.393542 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4e455-58eb-421d-9bc8-e784e806352d service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:57:59.393817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4e455-58eb-421d-9bc8-e784e806352d service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:57:59.394081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4e455-58eb-421d-9bc8-e784e806352d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:57:59.402296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b4e455-58eb-421d-9bc8-e784e806352d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:57:59.403697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b4e455-58eb-421d-9bc8-e784e806352d service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:57:59.413164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1053/2069] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:57:59 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:58:09.310630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:09.409000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=17 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:23.215006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b839235-2e28-4db2-98eb-e7eb3ea06fc6 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:23.287069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b839235-2e28-4db2-98eb-e7eb3ea06fc6 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:23.291680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:23.292341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:23.292624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:23.292900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] PATH_INFO: `/v3/users/0d7babaf2e3649d393478fa3ec9907ef` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:23.294131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d7babaf2e3649d393478fa3ec9907ef)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:23.296106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:23.350539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d7babaf2e3649d393478fa3ec9907ef'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.357594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d7babaf2e3649d393478fa3ec9907ef'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.372506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d7babaf2e3649d393478fa3ec9907ef'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.373428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d7babaf2e3649d393478fa3ec9907ef'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.374201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d7babaf2e3649d393478fa3ec9907ef'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.390615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1017/2070] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:58:23 2022] DELETE /identity/v3/users/0d7babaf2e3649d393478fa3ec9907ef => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:58:23.395099 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028533ad-d099-4b9f-a510-e4a8b3dbbbae None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:23.395633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028533ad-d099-4b9f-a510-e4a8b3dbbbae None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:23.395633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028533ad-d099-4b9f-a510-e4a8b3dbbbae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:23.416946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-028533ad-d099-4b9f-a510-e4a8b3dbbbae None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:23.418155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-028533ad-d099-4b9f-a510-e4a8b3dbbbae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:23.479253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1054/2071] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:58:23.490014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:23.491454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:23.551147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb88596a-1786-4449-a556-915b83148810 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:23.554775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f16b700-ab13-40f6-a629-9b110bbf7587 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:23.555403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16b700-ab13-40f6-a629-9b110bbf7587 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:23.555667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16b700-ab13-40f6-a629-9b110bbf7587 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:23.555942 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16b700-ab13-40f6-a629-9b110bbf7587 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:23.565565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f16b700-ab13-40f6-a629-9b110bbf7587 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:23.567004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f16b700-ab13-40f6-a629-9b110bbf7587 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:23.576563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1018/2072] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:58:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:58:23.848399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028533ad-d099-4b9f-a510-e4a8b3dbbbae None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:23.849830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-028533ad-d099-4b9f-a510-e4a8b3dbbbae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:23.862367 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028533ad-d099-4b9f-a510-e4a8b3dbbbae None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:23.867238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:23.867864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:23.868174 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:23.868448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] PATH_INFO: `/v3/projects/c722bfda48424b8fb600dd3f8150e29b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:23.869588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=c722bfda48424b8fb600dd3f8150e29b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:23.871484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:23.919325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c722bfda48424b8fb600dd3f8150e29b', 'request_id': 'req-78b665a4-5aee-483f-b04d-fef46c903f8f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.925731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c722bfda48424b8fb600dd3f8150e29b', 'request_id': 'req-78b665a4-5aee-483f-b04d-fef46c903f8f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.932740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c722bfda48424b8fb600dd3f8150e29b', 'request_id': 'req-78b665a4-5aee-483f-b04d-fef46c903f8f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.943527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] The token cache is being invalidate because project c722bfda48424b8fb600dd3f8150e29b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:58:23.943884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.944778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-78b665a4-5aee-483f-b04d-fef46c903f8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:23.946354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1055/2073] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:58:23 2022] DELETE /identity/v3/projects/c722bfda48424b8fb600dd3f8150e29b => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:58:23.951347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483fd7dc-8f31-4a2e-9e73-66544a707783 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:23.951635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483fd7dc-8f31-4a2e-9e73-66544a707783 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:23.951912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483fd7dc-8f31-4a2e-9e73-66544a707783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:23.972573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-483fd7dc-8f31-4a2e-9e73-66544a707783 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:23.973880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-483fd7dc-8f31-4a2e-9e73-66544a707783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:23.992471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:24.038201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1019/2074] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:58:24.046450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292a62d6-0da3-49f6-b65b-241e9b813285 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.046735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292a62d6-0da3-49f6-b65b-241e9b813285 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.047007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292a62d6-0da3-49f6-b65b-241e9b813285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.067180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-292a62d6-0da3-49f6-b65b-241e9b813285 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:24.068417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-292a62d6-0da3-49f6-b65b-241e9b813285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:24.076457 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1056/2075] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:58:24.081507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483fd7dc-8f31-4a2e-9e73-66544a707783 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:24.091931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483fd7dc-8f31-4a2e-9e73-66544a707783 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:24.097285 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:24.097887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.098168 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.098444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.099085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:58:24.099391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:24.101026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:24.113086 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1020/2076] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:58:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:58:24.117733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292a62d6-0da3-49f6-b65b-241e9b813285 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:24.127334 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292a62d6-0da3-49f6-b65b-241e9b813285 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:24.132258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2529a69e-28b2-4b7b-b207-8998dc94f3f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:24.132850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2529a69e-28b2-4b7b-b207-8998dc94f3f6 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.133128 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2529a69e-28b2-4b7b-b207-8998dc94f3f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.133397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2529a69e-28b2-4b7b-b207-8998dc94f3f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.134094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2529a69e-28b2-4b7b-b207-8998dc94f3f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:24.136014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2529a69e-28b2-4b7b-b207-8998dc94f3f6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:24.165598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1057/2077] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:58:24 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:58:24.169190 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:24.179182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70104728-9567-4cba-a3f3-c6ec8baa9a23 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:24.236508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc70336-6f4d-4fa4-9ae1-99fae6694c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:24.237211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc70336-6f4d-4fa4-9ae1-99fae6694c37 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.237454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc70336-6f4d-4fa4-9ae1-99fae6694c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.237725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc70336-6f4d-4fa4-9ae1-99fae6694c37 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.238377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc70336-6f4d-4fa4-9ae1-99fae6694c37 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:24.240664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc70336-6f4d-4fa4-9ae1-99fae6694c37 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:24.262003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1021/2078] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:58:24 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:58:24.266135 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2529a69e-28b2-4b7b-b207-8998dc94f3f6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:24.276884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2529a69e-28b2-4b7b-b207-8998dc94f3f6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:24.281398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d265a55-5683-4714-a95c-e813b0fa9242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:24.282011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d265a55-5683-4714-a95c-e813b0fa9242 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.282394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d265a55-5683-4714-a95c-e813b0fa9242 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.282669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d265a55-5683-4714-a95c-e813b0fa9242 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.283313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d265a55-5683-4714-a95c-e813b0fa9242 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:24.284928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d265a55-5683-4714-a95c-e813b0fa9242 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:24.295668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1058/2079] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:58:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:58:24.300180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc70336-6f4d-4fa4-9ae1-99fae6694c37 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:24.310281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc70336-6f4d-4fa4-9ae1-99fae6694c37 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:24.315924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d127f6-d569-4f9d-8718-5d30d72c0be2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:24.316613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d127f6-d569-4f9d-8718-5d30d72c0be2 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.316897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d127f6-d569-4f9d-8718-5d30d72c0be2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.317173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d127f6-d569-4f9d-8718-5d30d72c0be2 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.317810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d127f6-d569-4f9d-8718-5d30d72c0be2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:24.319432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d127f6-d569-4f9d-8718-5d30d72c0be2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:24.331206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1022/2080] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:58:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:58:24.335446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d265a55-5683-4714-a95c-e813b0fa9242 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:24.345585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d265a55-5683-4714-a95c-e813b0fa9242 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:24.350466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:24.351108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.351387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.351658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] PATH_INFO: `/v3/projects/f9a311649a4a42d2948beaf0ea57bf92/users/f6bb44dc46b94eada6d9ba18e432137a/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.366549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9a311649a4a42d2948beaf0ea57bf92, user_id=f6bb44dc46b94eada6d9ba18e432137a, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:24.369236 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:24.383079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1059/2081] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:58:24 2022] PUT /identity/v3/projects/f9a311649a4a42d2948beaf0ea57bf92/users/f6bb44dc46b94eada6d9ba18e432137a/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:58:24.395440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffbdf4-34b2-4407-9cfc-b2c22d7c64a5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.395731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffbdf4-34b2-4407-9cfc-b2c22d7c64a5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.396004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffbdf4-34b2-4407-9cfc-b2c22d7c64a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.423381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b8ffbdf4-34b2-4407-9cfc-b2c22d7c64a5 None None] MFA Rules not processed for user `f6bb44dc46b94eada6d9ba18e432137a`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:24.424665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b8ffbdf4-34b2-4407-9cfc-b2c22d7c64a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:24.497709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1023/2082] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:58:24.509848 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:24.511541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:24.534662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:24.571856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d847e16-10f4-4e82-9466-9eedc72708ee admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:24.575468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31e997-2afe-4df4-9a51-ea2bac6a0df0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:24.576065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31e997-2afe-4df4-9a51-ea2bac6a0df0 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:24.576370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31e997-2afe-4df4-9a51-ea2bac6a0df0 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:24.576642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31e997-2afe-4df4-9a51-ea2bac6a0df0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:24.585508 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31e997-2afe-4df4-9a51-ea2bac6a0df0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:24.586931 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31e997-2afe-4df4-9a51-ea2bac6a0df0 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:24.596573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1060/2083] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:58:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:58:25.206134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ffbdf4-34b2-4407-9cfc-b2c22d7c64a5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:25.216580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ffbdf4-34b2-4407-9cfc-b2c22d7c64a5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:25.220748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41033ef2-f5e3-4412-8b1f-a09afe7f8524 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:25.221350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41033ef2-f5e3-4412-8b1f-a09afe7f8524 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:25.221632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41033ef2-f5e3-4412-8b1f-a09afe7f8524 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:25.221910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41033ef2-f5e3-4412-8b1f-a09afe7f8524 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:25.223668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-41033ef2-f5e3-4412-8b1f-a09afe7f8524 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:25.238510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:25.285568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41033ef2-f5e3-4412-8b1f-a09afe7f8524 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:25.287002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41033ef2-f5e3-4412-8b1f-a09afe7f8524 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:25.296216 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1024/2084] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:58:25 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:58:27.166598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31e997-2afe-4df4-9a51-ea2bac6a0df0 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:27.237575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31e997-2afe-4df4-9a51-ea2bac6a0df0 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:27.243817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:27.244477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:27.244763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:27.245041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] PATH_INFO: `/v3/users/f6bb44dc46b94eada6d9ba18e432137a` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:27.246194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6bb44dc46b94eada6d9ba18e432137a)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:27.248122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:27.300708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6bb44dc46b94eada6d9ba18e432137a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:27.308599 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6bb44dc46b94eada6d9ba18e432137a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:27.320743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6bb44dc46b94eada6d9ba18e432137a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:27.321463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6bb44dc46b94eada6d9ba18e432137a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:27.322068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6bb44dc46b94eada6d9ba18e432137a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:27.335960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1061/2085] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:58:27 2022] DELETE /identity/v3/users/f6bb44dc46b94eada6d9ba18e432137a => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:58:27.340631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d782113-7de5-4e8e-9e91-a65b7c065de0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:27.341114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d782113-7de5-4e8e-9e91-a65b7c065de0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:27.341385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d782113-7de5-4e8e-9e91-a65b7c065de0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:27.361795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7d782113-7de5-4e8e-9e91-a65b7c065de0 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:27.363006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7d782113-7de5-4e8e-9e91-a65b7c065de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:27.424716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1025/2086] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:58:27.434286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:27.435687 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:27.483794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:27.498573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fff88b8-d096-4f0c-a441-6618ec9dbf94 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:27.503438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc88115e-dc52-40f9-9d17-88ece9b1b86b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:27.504039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc88115e-dc52-40f9-9d17-88ece9b1b86b service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:27.504341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc88115e-dc52-40f9-9d17-88ece9b1b86b service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:27.504612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc88115e-dc52-40f9-9d17-88ece9b1b86b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:27.513812 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc88115e-dc52-40f9-9d17-88ece9b1b86b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:27.515755 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc88115e-dc52-40f9-9d17-88ece9b1b86b service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:27.526494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1062/2087] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:58:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:58:27.979432 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d782113-7de5-4e8e-9e91-a65b7c065de0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:27.981268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7d782113-7de5-4e8e-9e91-a65b7c065de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:27.999603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d782113-7de5-4e8e-9e91-a65b7c065de0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:28.022293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:28.023025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.023373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.023715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] PATH_INFO: `/v3/projects/f9a311649a4a42d2948beaf0ea57bf92` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.024968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9a311649a4a42d2948beaf0ea57bf92)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:28.027881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:28.109996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9a311649a4a42d2948beaf0ea57bf92', 'request_id': 'req-65e9f278-2896-4f47-8657-0acb019871d6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:28.117143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9a311649a4a42d2948beaf0ea57bf92', 'request_id': 'req-65e9f278-2896-4f47-8657-0acb019871d6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:28.127368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9a311649a4a42d2948beaf0ea57bf92', 'request_id': 'req-65e9f278-2896-4f47-8657-0acb019871d6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:28.154313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] The token cache is being invalidate because project f9a311649a4a42d2948beaf0ea57bf92 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:58:28.154609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:28.155297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-65e9f278-2896-4f47-8657-0acb019871d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:58:28.161378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1026/2088] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:58:27 2022] DELETE /identity/v3/projects/f9a311649a4a42d2948beaf0ea57bf92 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:58:28.163064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576eccc-46df-47e3-87e7-2a5a1daeafd5 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.163434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576eccc-46df-47e3-87e7-2a5a1daeafd5 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.163760 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576eccc-46df-47e3-87e7-2a5a1daeafd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.191399 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-7576eccc-46df-47e3-87e7-2a5a1daeafd5 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:28.193522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7576eccc-46df-47e3-87e7-2a5a1daeafd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:28.284737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1063/2089] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:58:28.287964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cdf3c-6b11-4fb2-bd46-4785f8a438b0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.287964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cdf3c-6b11-4fb2-bd46-4785f8a438b0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.288277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cdf3c-6b11-4fb2-bd46-4785f8a438b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.310930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e80cdf3c-6b11-4fb2-bd46-4785f8a438b0 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:28.312182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e80cdf3c-6b11-4fb2-bd46-4785f8a438b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:28.321552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1027/2090] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:58:28.325792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7576eccc-46df-47e3-87e7-2a5a1daeafd5 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:28.337529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7576eccc-46df-47e3-87e7-2a5a1daeafd5 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:28.344697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:28.345515 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.345922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.346286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.347136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:58:28.347604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:28.349535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:28.370351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1064/2091] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 08:58:28.371371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cdf3c-6b11-4fb2-bd46-4785f8a438b0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:28.389155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cdf3c-6b11-4fb2-bd46-4785f8a438b0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:28.399494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaf823-d898-4ff9-8e7e-fb421c7c2015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:28.400946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaf823-d898-4ff9-8e7e-fb421c7c2015 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.401437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaf823-d898-4ff9-8e7e-fb421c7c2015 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.401798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaf823-d898-4ff9-8e7e-fb421c7c2015 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.403157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aaf823-d898-4ff9-8e7e-fb421c7c2015 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:28.405059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aaf823-d898-4ff9-8e7e-fb421c7c2015 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:28.441007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1028/2092] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:58:28 2022] POST /identity/v3/projects => generated 379 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:58:28.445292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:28.456750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9aa78-fe5a-4a18-9250-40582c58f7fa admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:28.547470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89368850-9725-443a-b5b9-b54dd49e0412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:28.548520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89368850-9725-443a-b5b9-b54dd49e0412 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.548878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89368850-9725-443a-b5b9-b54dd49e0412 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.549221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89368850-9725-443a-b5b9-b54dd49e0412 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.550139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89368850-9725-443a-b5b9-b54dd49e0412 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:28.552304 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89368850-9725-443a-b5b9-b54dd49e0412 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:28.582942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1065/2093] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:58:28 2022] POST /identity/v3/users => generated 290 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:58:28.584160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaf823-d898-4ff9-8e7e-fb421c7c2015 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:28.599193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaf823-d898-4ff9-8e7e-fb421c7c2015 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:28.608637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483e23d-3c6c-4234-a67e-c4115520486f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:28.609270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483e23d-3c6c-4234-a67e-c4115520486f admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.609550 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483e23d-3c6c-4234-a67e-c4115520486f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.610644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483e23d-3c6c-4234-a67e-c4115520486f admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.610644 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2483e23d-3c6c-4234-a67e-c4115520486f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:28.612516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2483e23d-3c6c-4234-a67e-c4115520486f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:28.624455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:28.630866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1029/2094] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:58:28.635492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89368850-9725-443a-b5b9-b54dd49e0412 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:28.647524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89368850-9725-443a-b5b9-b54dd49e0412 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:28.654303 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff14146b-ca6d-46dc-8a3b-7bbc5991bdd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:28.655726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff14146b-ca6d-46dc-8a3b-7bbc5991bdd7 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.656148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff14146b-ca6d-46dc-8a3b-7bbc5991bdd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.658140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff14146b-ca6d-46dc-8a3b-7bbc5991bdd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.658842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff14146b-ca6d-46dc-8a3b-7bbc5991bdd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:28.661423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff14146b-ca6d-46dc-8a3b-7bbc5991bdd7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:28.678386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1066/2095] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:58:28.682734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483e23d-3c6c-4234-a67e-c4115520486f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:28.694024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483e23d-3c6c-4234-a67e-c4115520486f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:28.699144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:28.699807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.700171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.700424 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] PATH_INFO: `/v3/projects/01318d53c6ed499e94a25e12f826719f/users/c94f6e2fa2b94630a745af7c54ec680a/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.720044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] RBAC: Authorizing `identity:create_grant(project_id=01318d53c6ed499e94a25e12f826719f, user_id=c94f6e2fa2b94630a745af7c54ec680a, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:28.723353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:28.738486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1030/2096] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:58:28 2022] PUT /identity/v3/projects/01318d53c6ed499e94a25e12f826719f/users/c94f6e2fa2b94630a745af7c54ec680a/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:58:28.743946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943a54a5-f168-430b-b079-d2a2e92045c2 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.744272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943a54a5-f168-430b-b079-d2a2e92045c2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.744543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943a54a5-f168-430b-b079-d2a2e92045c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.766782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-943a54a5-f168-430b-b079-d2a2e92045c2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:28.768308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-943a54a5-f168-430b-b079-d2a2e92045c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:28.817526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:28.834423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1067/2097] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:58:28.843714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:28.845511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:28.904159 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d95f7-1bca-4181-8452-eb87f6b3225d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:28.907873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:28.908676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:28.908961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:28.909237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:28.919714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:28.921194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:28.931149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1031/2098] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:58:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:58:29.625045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75f1d4-257e-47b6-960f-7093732ac401 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:29.625724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75f1d4-257e-47b6-960f-7093732ac401 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:29.625724 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75f1d4-257e-47b6-960f-7093732ac401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:29.653942 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6b75f1d4-257e-47b6-960f-7093732ac401 None None] MFA Rules not processed for user `c94f6e2fa2b94630a745af7c54ec680a`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:29.655245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6b75f1d4-257e-47b6-960f-7093732ac401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:29.723470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1068/2099] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:29 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:58:29.734420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:58:29.736543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:29.794384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dce46d-6a50-4833-86fe-a260de99284c service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:58:29.798320 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86594bd4-8cf1-4ced-93e9-279188a01773 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:58:29.798927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86594bd4-8cf1-4ced-93e9-279188a01773 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:29.799207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86594bd4-8cf1-4ced-93e9-279188a01773 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:29.799480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86594bd4-8cf1-4ced-93e9-279188a01773 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:29.809348 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86594bd4-8cf1-4ced-93e9-279188a01773 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:58:29.810850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86594bd4-8cf1-4ced-93e9-279188a01773 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:58:29.820367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1032/2100] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:58:29 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:58:31.210347 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dd06e-ed8c-4291-b95d-3d4a2f054ba5 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:31.210938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dd06e-ed8c-4291-b95d-3d4a2f054ba5 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:31.210938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dd06e-ed8c-4291-b95d-3d4a2f054ba5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:31.232290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-871dd06e-ed8c-4291-b95d-3d4a2f054ba5 None None] MFA Rules not processed for user `c94f6e2fa2b94630a745af7c54ec680a`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:31.233604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-871dd06e-ed8c-4291-b95d-3d4a2f054ba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:31.243002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1069/2101] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:58:32.266886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffa634f-d69f-4d03-b485-0e68e5d7ffde None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:58:32.267653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffa634f-d69f-4d03-b485-0e68e5d7ffde None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:58:32.268282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffa634f-d69f-4d03-b485-0e68e5d7ffde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:58:32.290288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-bffa634f-d69f-4d03-b485-0e68e5d7ffde None None] MFA Rules not processed for user `94169e17266b4e6fad7e068a4c1384c7`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:58:32.291613 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-bffa634f-d69f-4d03-b485-0e68e5d7ffde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:58:32.365001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1033/2102] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:58:32 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:58:41.226321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=39 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:58:42.346458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=87,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:00.448682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe41abec-b2d9-44cd-bd60-e4a256b3ec63 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:00.449215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe41abec-b2d9-44cd-bd60-e4a256b3ec63 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:00.449215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe41abec-b2d9-44cd-bd60-e4a256b3ec63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:00.478131 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fe41abec-b2d9-44cd-bd60-e4a256b3ec63 None None] MFA Rules not processed for user `ebdef1d69bb14837b9a0c28cbba332fc`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:59:00.479353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fe41abec-b2d9-44cd-bd60-e4a256b3ec63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:00.553670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1070/2103] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:59:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:59:10.551850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=37 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:13.330202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f39e8-6f8b-477e-aeec-70c813cc6c26 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:13.330685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f39e8-6f8b-477e-aeec-70c813cc6c26 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:13.330779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f39e8-6f8b-477e-aeec-70c813cc6c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:13.361327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-638f39e8-6f8b-477e-aeec-70c813cc6c26 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:59:13.362540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-638f39e8-6f8b-477e-aeec-70c813cc6c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:13.441311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1034/2104] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:59:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:59:13.487134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe41abec-b2d9-44cd-bd60-e4a256b3ec63 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:13.488863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fe41abec-b2d9-44cd-bd60-e4a256b3ec63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:13.567151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe41abec-b2d9-44cd-bd60-e4a256b3ec63 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:13.572427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:13.573035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:13.573316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:13.573592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:13.582293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:13.583895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:13.592110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1071/2105] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:59:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:59:23.434469 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=39 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:23.591268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=35 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:28.289405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638f39e8-6f8b-477e-aeec-70c813cc6c26 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:28.290914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-638f39e8-6f8b-477e-aeec-70c813cc6c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:28.325589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638f39e8-6f8b-477e-aeec-70c813cc6c26 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:28.330079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c902fa5-985d-443e-a9d3-30c45d3315ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:28.330723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c902fa5-985d-443e-a9d3-30c45d3315ae service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:28.331070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c902fa5-985d-443e-a9d3-30c45d3315ae service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:28.331463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c902fa5-985d-443e-a9d3-30c45d3315ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:28.333247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4c902fa5-985d-443e-a9d3-30c45d3315ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:28.396761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c902fa5-985d-443e-a9d3-30c45d3315ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:28.398204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c902fa5-985d-443e-a9d3-30c45d3315ae service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:28.409300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1035/2106] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:59:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:59:36.236972 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:36.238361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:36.330155 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004d22ed-0b12-4430-9d4b-b790859c1810 service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:36.334933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb378c1-7532-44a3-a023-5b94e795acf4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:36.335525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb378c1-7532-44a3-a023-5b94e795acf4 service placement] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:36.335800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb378c1-7532-44a3-a023-5b94e795acf4 service placement] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:36.336068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb378c1-7532-44a3-a023-5b94e795acf4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:36.337850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fbb378c1-7532-44a3-a023-5b94e795acf4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:36.350168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb378c1-7532-44a3-a023-5b94e795acf4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:36.351714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb378c1-7532-44a3-a023-5b94e795acf4 service placement] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:36.362276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1072/2107] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:59:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:59:38.323906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87d549-f2a8-4c30-b789-ad066f9b0931 None None] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:38.324485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87d549-f2a8-4c30-b789-ad066f9b0931 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:38.324595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87d549-f2a8-4c30-b789-ad066f9b0931 None None] PATH_INFO: `/` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:38.325857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1036/2108] 10.222.0.39 () {54 vars in 921 bytes} [Mon Oct 24 08:59:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 24 08:59:38.331512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99baef6-cf42-4f3b-b039-2aeb1c75c5fd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:38.332000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99baef6-cf42-4f3b-b039-2aeb1c75c5fd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:38.332111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99baef6-cf42-4f3b-b039-2aeb1c75c5fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:38.377830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c99baef6-cf42-4f3b-b039-2aeb1c75c5fd None None] MFA Rules not processed for user `4244858dfb464d029b6f6bed0491557d`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:59:38.378982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c99baef6-cf42-4f3b-b039-2aeb1c75c5fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:38.403801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=37 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:38.466725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1073/2109] 10.222.0.39 () {58 vars in 1018 bytes} [Mon Oct 24 08:59:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 08:59:38.478756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d87d549-f2a8-4c30-b789-ad066f9b0931 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:38.480482 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4d87d549-f2a8-4c30-b789-ad066f9b0931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:38.537665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d87d549-f2a8-4c30-b789-ad066f9b0931 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:38.541247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8126ff3-935c-426a-8e88-9282099e467e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a4fd11d5d85479c9aa1d87b23a13141', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:38.541843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8126ff3-935c-426a-8e88-9282099e467e service glance] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:38.542119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8126ff3-935c-426a-8e88-9282099e467e service glance] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:38.542384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8126ff3-935c-426a-8e88-9282099e467e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:38.555148 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8126ff3-935c-426a-8e88-9282099e467e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:38.556634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8126ff3-935c-426a-8e88-9282099e467e service glance] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:38.567599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1037/2110] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:59:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:59:39.553503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99baef6-cf42-4f3b-b039-2aeb1c75c5fd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:39.566379 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99baef6-cf42-4f3b-b039-2aeb1c75c5fd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:39.569950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:39.570610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:39.570886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:39.571156 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:39.572937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:39.607797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:39.609271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:39.615287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:39.631837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1074/2111] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:59:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:59:42.937799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df841b40-f8f8-4b9f-8d92-cd92aed75e77 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:42.938382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df841b40-f8f8-4b9f-8d92-cd92aed75e77 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:42.939029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df841b40-f8f8-4b9f-8d92-cd92aed75e77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:42.976630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-df841b40-f8f8-4b9f-8d92-cd92aed75e77 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:59:42.978353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-df841b40-f8f8-4b9f-8d92-cd92aed75e77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:42.989258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1038/2112] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 08:59:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:59:43.010549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:43.030064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85459115-bc33-47c1-be9f-c3faf92fa4c3 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:43.038750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0142b68-a7a1-48ff-9658-5ac8c3ddf384 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:43.039800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0142b68-a7a1-48ff-9658-5ac8c3ddf384 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:43.040340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0142b68-a7a1-48ff-9658-5ac8c3ddf384 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:43.041072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0142b68-a7a1-48ff-9658-5ac8c3ddf384 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:43.053273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0142b68-a7a1-48ff-9658-5ac8c3ddf384 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:43.055752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0142b68-a7a1-48ff-9658-5ac8c3ddf384 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:43.073497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1075/2113] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 08:59:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 08:59:44.300656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df841b40-f8f8-4b9f-8d92-cd92aed75e77 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:44.302264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-df841b40-f8f8-4b9f-8d92-cd92aed75e77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:44.318714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df841b40-f8f8-4b9f-8d92-cd92aed75e77 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:44.324675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:44.325733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:44.326358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:44.326852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:44.329122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:44.347553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:44.353987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:44.380241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1039/2114] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:59:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:59:48.335954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0142b68-a7a1-48ff-9658-5ac8c3ddf384 service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:48.362020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0142b68-a7a1-48ff-9658-5ac8c3ddf384 service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:48.375345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:48.375987 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:48.376301 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:48.376574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:48.379110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:48.420723 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:48.422739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:48.423568 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:48.425305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:48.457779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8526d5-064e-4770-9b5c-253ee9db05e4 service placement] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:48.462349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1076/2115] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:59:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:59:48.467973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:48.469739 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:48.470235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:48.471211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] PATH_INFO: `/v3/users/82055ff074d842e88e27a441c32e9d01` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:48.473063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82055ff074d842e88e27a441c32e9d01)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:48.478433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:48.556670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82055ff074d842e88e27a441c32e9d01'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.566565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82055ff074d842e88e27a441c32e9d01'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.582517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82055ff074d842e88e27a441c32e9d01'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.587179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82055ff074d842e88e27a441c32e9d01'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.588116 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82055ff074d842e88e27a441c32e9d01'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.606376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1040/2116] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:59:48 2022] DELETE /identity/v3/users/82055ff074d842e88e27a441c32e9d01 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:59:48.813005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:48.829093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6220e5e4-4ea0-42c3-a481-9803491c239a service placement] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:48.908559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:48.909325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:48.909815 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:48.910089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] PATH_INFO: `/v3/projects/29e4576f2b744d40878d65470e24c24c` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:48.911319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=29e4576f2b744d40878d65470e24c24c)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:48.913267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:48.966992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29e4576f2b744d40878d65470e24c24c', 'request_id': 'req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.974674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29e4576f2b744d40878d65470e24c24c', 'request_id': 'req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.983134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29e4576f2b744d40878d65470e24c24c', 'request_id': 'req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.995072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] The token cache is being invalidate because project 29e4576f2b744d40878d65470e24c24c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:59:48.995441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.996188 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:48.997804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1077/2117] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:59:48 2022] DELETE /identity/v3/projects/29e4576f2b744d40878d65470e24c24c => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:59:49.002221 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:49.004461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:49.019544 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7c648-aaab-41a4-a74d-2e8801475629 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:49.079184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:49.088286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:49.088933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:49.089210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:49.089710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] PATH_INFO: `/v3/users/dbbaeca00feb411b84c495b267a618e8` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:49.090955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbbaeca00feb411b84c495b267a618e8)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:49.094521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:49.179987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbbaeca00feb411b84c495b267a618e8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.189826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbbaeca00feb411b84c495b267a618e8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.210264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbbaeca00feb411b84c495b267a618e8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.212651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbbaeca00feb411b84c495b267a618e8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.216719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-41b73ce4-1552-4510-8b4a-e99da15eb987 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbbaeca00feb411b84c495b267a618e8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.233675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1041/2118] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 08:59:49 2022] DELETE /identity/v3/users/dbbaeca00feb411b84c495b267a618e8 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:59:49.461212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:49.463240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:49.486837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9ced6e-2665-4fc2-9c34-a119a7e40f2c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:49.554557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:49.563819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:49.564632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:49.565286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:49.565596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] PATH_INFO: `/v3/projects/0c91fbd0e15a466a894e90236b7c0f90` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:49.566853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c91fbd0e15a466a894e90236b7c0f90)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:49.569375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:49.619660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c91fbd0e15a466a894e90236b7c0f90', 'request_id': 'req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.627442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c91fbd0e15a466a894e90236b7c0f90', 'request_id': 'req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.636676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c91fbd0e15a466a894e90236b7c0f90', 'request_id': 'req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.650617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] The token cache is being invalidate because project 0c91fbd0e15a466a894e90236b7c0f90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 08:59:49.651091 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.652043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f24680cf-5ef3-41fd-9a51-f0f6ab893b86 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 08:59:49.653601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1078/2119] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 08:59:49 2022] DELETE /identity/v3/projects/0c91fbd0e15a466a894e90236b7c0f90 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:59:49.659020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239a655-d3f6-4d46-b3fe-41f5689d17a7 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:49.659523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239a655-d3f6-4d46-b3fe-41f5689d17a7 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:49.659756 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239a655-d3f6-4d46-b3fe-41f5689d17a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:49.726736 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f239a655-d3f6-4d46-b3fe-41f5689d17a7 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:59:49.727946 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f239a655-d3f6-4d46-b3fe-41f5689d17a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:49.802671 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1042/2120] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:59:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:59:49.811658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd8600-9736-4dba-90b6-cc9b63e7992a None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:49.811948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd8600-9736-4dba-90b6-cc9b63e7992a None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:49.812254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd8600-9736-4dba-90b6-cc9b63e7992a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:49.854787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-12fd8600-9736-4dba-90b6-cc9b63e7992a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:59:49.856426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-12fd8600-9736-4dba-90b6-cc9b63e7992a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:49.872255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1079/2121] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:59:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:59:49.876972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f239a655-d3f6-4d46-b3fe-41f5689d17a7 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:49.888208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f239a655-d3f6-4d46-b3fe-41f5689d17a7 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:49.893573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:49.894329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:49.894611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:49.894929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:49.895553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 08:59:49.895856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:49.897525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:49.914143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1043/2122] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 08:59:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 08:59:49.915368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fd8600-9736-4dba-90b6-cc9b63e7992a None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:49.927851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fd8600-9736-4dba-90b6-cc9b63e7992a None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:49.935341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0497875f-9ed1-4587-8e64-6a36c9c066c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:49.936960 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0497875f-9ed1-4587-8e64-6a36c9c066c0 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:49.937537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0497875f-9ed1-4587-8e64-6a36c9c066c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:49.937997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0497875f-9ed1-4587-8e64-6a36c9c066c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:49.938827 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0497875f-9ed1-4587-8e64-6a36c9c066c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:49.940792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0497875f-9ed1-4587-8e64-6a36c9c066c0 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:49.971268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1080/2123] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:59:49 2022] POST /identity/v3/projects => generated 379 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 08:59:49.977241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:49.996610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a77ec4-73a7-48bd-8688-3c0136eac570 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:50.039454 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:50.065829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3d203-54e4-4858-94a1-d74d5d81b2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:50.066555 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3d203-54e4-4858-94a1-d74d5d81b2c4 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:50.066845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3d203-54e4-4858-94a1-d74d5d81b2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:50.067121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3d203-54e4-4858-94a1-d74d5d81b2c4 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:50.067760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a3d203-54e4-4858-94a1-d74d5d81b2c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:50.069768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a3d203-54e4-4858-94a1-d74d5d81b2c4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:50.093641 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1044/2124] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:59:49 2022] POST /identity/v3/users => generated 290 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:59:50.097318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0497875f-9ed1-4587-8e64-6a36c9c066c0 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:50.109807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0497875f-9ed1-4587-8e64-6a36c9c066c0 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:50.114904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0de55-8306-4d7a-b02e-95e96a013f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:50.115547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0de55-8306-4d7a-b02e-95e96a013f5b admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:50.115839 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0de55-8306-4d7a-b02e-95e96a013f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:50.116114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0de55-8306-4d7a-b02e-95e96a013f5b admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:50.116796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f0de55-8306-4d7a-b02e-95e96a013f5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:50.118403 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f0de55-8306-4d7a-b02e-95e96a013f5b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:50.130436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1081/2125] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:59:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:59:50.134219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3d203-54e4-4858-94a1-d74d5d81b2c4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:50.145875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3d203-54e4-4858-94a1-d74d5d81b2c4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:50.150568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427d42b7-8705-4779-812b-5b3f85327471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:50.151169 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d42b7-8705-4779-812b-5b3f85327471 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:50.151455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d42b7-8705-4779-812b-5b3f85327471 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:50.151729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d42b7-8705-4779-812b-5b3f85327471 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:50.152406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427d42b7-8705-4779-812b-5b3f85327471 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:50.154572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427d42b7-8705-4779-812b-5b3f85327471 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:50.165588 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1045/2126] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:59:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:59:50.170414 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0de55-8306-4d7a-b02e-95e96a013f5b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:50.181258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0de55-8306-4d7a-b02e-95e96a013f5b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:50.187197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:50.187864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:50.188157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:50.188437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] PATH_INFO: `/v3/projects/30742e7b914243df8dd120079560a3de/users/3cddd27c37514e74a38430c46cef16fa/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:50.205791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] RBAC: Authorizing `identity:create_grant(project_id=30742e7b914243df8dd120079560a3de, user_id=3cddd27c37514e74a38430c46cef16fa, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:50.208995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:50.228677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1082/2127] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:59:50 2022] PUT /identity/v3/projects/30742e7b914243df8dd120079560a3de/users/3cddd27c37514e74a38430c46cef16fa/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 08:59:50.230139 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f4347-77c3-498e-a2d0-b77cafd2e456 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:50.230435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f4347-77c3-498e-a2d0-b77cafd2e456 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:50.230710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f4347-77c3-498e-a2d0-b77cafd2e456 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:50.251733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-1a9f4347-77c3-498e-a2d0-b77cafd2e456 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:59:50.253012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1a9f4347-77c3-498e-a2d0-b77cafd2e456 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:50.325243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1046/2128] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:59:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:59:50.335158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:50.336986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:50.413883 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93a9199-067a-48b4-9e31-9d17d50a399a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:50.417805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce938ed1-9b14-49ba-b6fc-06904728014b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:50.418448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce938ed1-9b14-49ba-b6fc-06904728014b service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:50.418761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce938ed1-9b14-49ba-b6fc-06904728014b service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:50.419007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce938ed1-9b14-49ba-b6fc-06904728014b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:50.429410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce938ed1-9b14-49ba-b6fc-06904728014b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:50.431019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce938ed1-9b14-49ba-b6fc-06904728014b service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:50.443220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1083/2129] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 08:59:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 08:59:55.297027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a7c842-d20b-4337-8401-67f80ee90f1f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:55.297567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a7c842-d20b-4337-8401-67f80ee90f1f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:55.297567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a7c842-d20b-4337-8401-67f80ee90f1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:55.331381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-31a7c842-d20b-4337-8401-67f80ee90f1f None None] MFA Rules not processed for user `3cddd27c37514e74a38430c46cef16fa`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 08:59:55.332666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-31a7c842-d20b-4337-8401-67f80ee90f1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 08:59:55.415095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1047/2130] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 08:59:55 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 08:59:55.419737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce938ed1-9b14-49ba-b6fc-06904728014b service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:55.430517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce938ed1-9b14-49ba-b6fc-06904728014b service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:55.435950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24c4a8b-35c8-42b7-a6b2-a3fa76c4ee58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:55.436593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c4a8b-35c8-42b7-a6b2-a3fa76c4ee58 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:55.436873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c4a8b-35c8-42b7-a6b2-a3fa76c4ee58 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:55.437144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c4a8b-35c8-42b7-a6b2-a3fa76c4ee58 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:55.437777 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24c4a8b-35c8-42b7-a6b2-a3fa76c4ee58 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:55.439683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24c4a8b-35c8-42b7-a6b2-a3fa76c4ee58 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:55.471992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1084/2131] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 08:59:55 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:59:55.475857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a7c842-d20b-4337-8401-67f80ee90f1f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:55.495729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a7c842-d20b-4337-8401-67f80ee90f1f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:55.574198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:55.591297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df1a342-3f87-4247-91d1-ffd4a7a811fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:55.592214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1a342-3f87-4247-91d1-ffd4a7a811fe admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:55.592486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1a342-3f87-4247-91d1-ffd4a7a811fe admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:55.592760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1a342-3f87-4247-91d1-ffd4a7a811fe admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:55.593585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df1a342-3f87-4247-91d1-ffd4a7a811fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:55.595562 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df1a342-3f87-4247-91d1-ffd4a7a811fe admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:55.625471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1048/2132] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 08:59:55 2022] POST /identity/v3/users => generated 296 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 08:59:55.629720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24c4a8b-35c8-42b7-a6b2-a3fa76c4ee58 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:55.635570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 08:59:55.643352 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24c4a8b-35c8-42b7-a6b2-a3fa76c4ee58 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:55.650761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401f6f96-6363-43d8-b1dd-2c49188bd335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:55.651561 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f6f96-6363-43d8-b1dd-2c49188bd335 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:55.651943 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f6f96-6363-43d8-b1dd-2c49188bd335 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:55.652358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f6f96-6363-43d8-b1dd-2c49188bd335 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:55.653108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401f6f96-6363-43d8-b1dd-2c49188bd335 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:55.655671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401f6f96-6363-43d8-b1dd-2c49188bd335 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:55.672172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1085/2133] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:59:55 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:59:55.673474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df1a342-3f87-4247-91d1-ffd4a7a811fe admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:55.686332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df1a342-3f87-4247-91d1-ffd4a7a811fe admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:55.691374 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbb2f6f-6983-4498-b800-161f3cac7191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:55.692039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb2f6f-6983-4498-b800-161f3cac7191 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:55.692382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb2f6f-6983-4498-b800-161f3cac7191 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:55.692663 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb2f6f-6983-4498-b800-161f3cac7191 admin admin] PATH_INFO: `/v3/projects/c7a6f9b4599c433fa5666799780fbadb/users/0d7d316d62334b55a73221a3cb488c31/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:55.710534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbb2f6f-6983-4498-b800-161f3cac7191 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7a6f9b4599c433fa5666799780fbadb, user_id=0d7d316d62334b55a73221a3cb488c31, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:55.713246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbb2f6f-6983-4498-b800-161f3cac7191 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:55.732653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1049/2134] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:59:55 2022] PUT /identity/v3/projects/c7a6f9b4599c433fa5666799780fbadb/users/0d7d316d62334b55a73221a3cb488c31/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 08:59:55.737021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401f6f96-6363-43d8-b1dd-2c49188bd335 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:55.747605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401f6f96-6363-43d8-b1dd-2c49188bd335 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:55.811469 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2035f12-5b74-4b68-a6e5-86f7a0e8780b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:55.812292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2035f12-5b74-4b68-a6e5-86f7a0e8780b admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:55.812570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2035f12-5b74-4b68-a6e5-86f7a0e8780b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:55.812877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2035f12-5b74-4b68-a6e5-86f7a0e8780b admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:55.813605 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2035f12-5b74-4b68-a6e5-86f7a0e8780b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:55.815274 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2035f12-5b74-4b68-a6e5-86f7a0e8780b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:55.830193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1086/2135] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 08:59:55 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 08:59:55.831439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbb2f6f-6983-4498-b800-161f3cac7191 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 08:59:55.843108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbb2f6f-6983-4498-b800-161f3cac7191 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 08:59:55.848152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 08:59:55.848950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 08:59:55.849228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 08:59:55.849502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] PATH_INFO: `/v3/projects/c7a6f9b4599c433fa5666799780fbadb/users/0d7d316d62334b55a73221a3cb488c31/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 08:59:55.851608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7a6f9b4599c433fa5666799780fbadb, user_id=0d7d316d62334b55a73221a3cb488c31, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 08:59:55.854509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 08:59:55.867614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1050/2136] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 08:59:55 2022] PUT /identity/v3/projects/c7a6f9b4599c433fa5666799780fbadb/users/0d7d316d62334b55a73221a3cb488c31/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:00:00.848545 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722a0b8-caa2-47d9-9043-4eda202efa71 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:00.849062 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722a0b8-caa2-47d9-9043-4eda202efa71 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:00.849162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722a0b8-caa2-47d9-9043-4eda202efa71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:00.879648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e722a0b8-caa2-47d9-9043-4eda202efa71 None None] MFA Rules not processed for user `0d7d316d62334b55a73221a3cb488c31`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:00:00.880709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e722a0b8-caa2-47d9-9043-4eda202efa71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:00.985795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1087/2137] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:00:00 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:00:01.002257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:00:01.004077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:01.105505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9c064-e4c0-42e0-9d5e-d53abe1832bc admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:00:01.114062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cd0c05-9057-434f-90bf-0e26d577c13d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:00:01.116017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cd0c05-9057-434f-90bf-0e26d577c13d service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:01.116392 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cd0c05-9057-434f-90bf-0e26d577c13d service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:01.116587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cd0c05-9057-434f-90bf-0e26d577c13d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:01.186458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cd0c05-9057-434f-90bf-0e26d577c13d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:00:01.188061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cd0c05-9057-434f-90bf-0e26d577c13d service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:00:01.226066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1051/2138] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:00:01 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:00:10.976716 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=67 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:00:11.215056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=78,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:00:14.957175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb7b13-215b-47dd-abb3-f71cc72f2a41 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:14.957752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb7b13-215b-47dd-abb3-f71cc72f2a41 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:14.957752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb7b13-215b-47dd-abb3-f71cc72f2a41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:14.977432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-dacb7b13-215b-47dd-abb3-f71cc72f2a41 None None] MFA Rules not processed for user `c94f6e2fa2b94630a745af7c54ec680a`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:00:14.978683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-dacb7b13-215b-47dd-abb3-f71cc72f2a41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:15.048060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1088/2139] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:00:14 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:00:15.848189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a92ea0b-0992-4681-ad9a-e77a762ee5cd None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:15.849025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a92ea0b-0992-4681-ad9a-e77a762ee5cd None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:15.849324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a92ea0b-0992-4681-ad9a-e77a762ee5cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:15.875995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-4a92ea0b-0992-4681-ad9a-e77a762ee5cd None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:00:15.877438 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4a92ea0b-0992-4681-ad9a-e77a762ee5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:15.949155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1052/2140] 10.222.0.39 () {58 vars in 967 bytes} [Mon Oct 24 09:00:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:00:15.978843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacb7b13-215b-47dd-abb3-f71cc72f2a41 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:00:15.984338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-dacb7b13-215b-47dd-abb3-f71cc72f2a41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:16.074210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacb7b13-215b-47dd-abb3-f71cc72f2a41 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:00:16.081653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23013dfb-ae0b-408f-8c43-8e51660ae24b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:00:16.082749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23013dfb-ae0b-408f-8c43-8e51660ae24b service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:16.083044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23013dfb-ae0b-408f-8c43-8e51660ae24b service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:16.083521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23013dfb-ae0b-408f-8c43-8e51660ae24b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:16.095943 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23013dfb-ae0b-408f-8c43-8e51660ae24b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:00:16.097581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23013dfb-ae0b-408f-8c43-8e51660ae24b service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:00:16.109211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1089/2141] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 09:00:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 09:00:25.936605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=25 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:00:26.103804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=57 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:00:31.557679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883015cc-0f4e-4f80-b5b8-610c287ab7cb None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:31.558197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883015cc-0f4e-4f80-b5b8-610c287ab7cb None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:31.558316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883015cc-0f4e-4f80-b5b8-610c287ab7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:31.579383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-883015cc-0f4e-4f80-b5b8-610c287ab7cb None None] MFA Rules not processed for user `3cddd27c37514e74a38430c46cef16fa`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:00:31.580549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-883015cc-0f4e-4f80-b5b8-610c287ab7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:31.587351 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1053/2142] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:00:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:00:41.572279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:00:51.109569 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23013dfb-ae0b-408f-8c43-8e51660ae24b service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:00:51.164867 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23013dfb-ae0b-408f-8c43-8e51660ae24b service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:00:51.169616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:00:51.170207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:51.170482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:51.170748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:51.172485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:51.185648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:00:51.187056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:00:51.197644 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1090/2143] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:00:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:00:53.569070 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883015cc-0f4e-4f80-b5b8-610c287ab7cb None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:00:53.579904 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883015cc-0f4e-4f80-b5b8-610c287ab7cb None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:00:53.583437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:00:53.584026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:53.584334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:53.584611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:53.586363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:53.600382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:00:53.601798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:00:53.612654 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1054/2144] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:00:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:00:58.669421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:00:58.671338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:00:58.683230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a4c4d4-43b5-4af3-b296-01e1ce1815d8 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:00:58.687823 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:00:58.688468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:00:58.688749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:00:58.689020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:00:58.698407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:00:58.699975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:00:58.709953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1091/2145] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:00:58 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:03.607750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=6 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:05.563880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:05.572433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfed644-6c03-4797-a00a-1182c7408f13 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:05.576659 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:05.577268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:05.577543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:05.577821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:05.579791 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:05.591373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:05.592911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:05.601770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1055/2146] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:08.705551 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=32 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:15.597980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=6 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:26.125751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:26.127177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:26.195265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02c1ae9-d152-4b87-9f42-0471839b63ef service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:26.199740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:26.200463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:26.200679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:26.200929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] PATH_INFO: `/v3/users/c94f6e2fa2b94630a745af7c54ec680a` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:26.202026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c94f6e2fa2b94630a745af7c54ec680a)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:26.203908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:26.252064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c94f6e2fa2b94630a745af7c54ec680a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.259183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c94f6e2fa2b94630a745af7c54ec680a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.271194 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c94f6e2fa2b94630a745af7c54ec680a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.271889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c94f6e2fa2b94630a745af7c54ec680a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.272521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ed278463-0207-4a57-83ed-718f6fa5328f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c94f6e2fa2b94630a745af7c54ec680a'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.285896 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1092/2147] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:01:26 2022] DELETE /identity/v3/users/c94f6e2fa2b94630a745af7c54ec680a => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:26.457162 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:26.474366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40347a8e-4668-49cc-8005-1d22ce3233bb service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:26.533277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:26.533898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:26.534177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:26.534450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] PATH_INFO: `/v3/projects/01318d53c6ed499e94a25e12f826719f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:26.535570 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] RBAC: Authorizing `identity:delete_project(project_id=01318d53c6ed499e94a25e12f826719f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:26.537539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:26.583156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01318d53c6ed499e94a25e12f826719f', 'request_id': 'req-76ade26a-232f-460b-a5cb-079ebb6c023b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.588471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01318d53c6ed499e94a25e12f826719f', 'request_id': 'req-76ade26a-232f-460b-a5cb-079ebb6c023b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.594935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01318d53c6ed499e94a25e12f826719f', 'request_id': 'req-76ade26a-232f-460b-a5cb-079ebb6c023b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.607857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] The token cache is being invalidate because project 01318d53c6ed499e94a25e12f826719f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:01:26.608241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.608909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-76ade26a-232f-460b-a5cb-079ebb6c023b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:26.610394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1056/2148] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:01:26 2022] DELETE /identity/v3/projects/01318d53c6ed499e94a25e12f826719f => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:26.615821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e33644-7f64-45db-b46b-dcc1a90f3d4f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:26.616110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e33644-7f64-45db-b46b-dcc1a90f3d4f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:26.616415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e33644-7f64-45db-b46b-dcc1a90f3d4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:26.636468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e9e33644-7f64-45db-b46b-dcc1a90f3d4f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:26.637676 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e9e33644-7f64-45db-b46b-dcc1a90f3d4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:26.701574 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1093/2149] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:26.709928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f7d70-5700-4d5d-b579-7c359827d27a None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:26.710231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f7d70-5700-4d5d-b579-7c359827d27a None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:26.710503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f7d70-5700-4d5d-b579-7c359827d27a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:26.730992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-384f7d70-5700-4d5d-b579-7c359827d27a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:26.732268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-384f7d70-5700-4d5d-b579-7c359827d27a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:26.740971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1057/2150] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:26.747050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e33644-7f64-45db-b46b-dcc1a90f3d4f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:26.757206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e33644-7f64-45db-b46b-dcc1a90f3d4f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:26.763218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:26.763815 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:26.764090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:26.764389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:26.765040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:01:26.765386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:26.766993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:26.783834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1094/2151] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:01:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 09:01:26.785025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384f7d70-5700-4d5d-b579-7c359827d27a None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:26.794725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384f7d70-5700-4d5d-b579-7c359827d27a None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:26.799998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5159e3e-35cf-41b5-8507-992733cbe540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:26.800676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5159e3e-35cf-41b5-8507-992733cbe540 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:26.800958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5159e3e-35cf-41b5-8507-992733cbe540 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:26.801234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5159e3e-35cf-41b5-8507-992733cbe540 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:26.801886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5159e3e-35cf-41b5-8507-992733cbe540 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:26.803848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5159e3e-35cf-41b5-8507-992733cbe540 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:26.853630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1058/2152] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:01:26 2022] POST /identity/v3/projects => generated 383 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:01:26.856580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:26.868300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c482336e-bf99-4795-8f71-e1ba4a77c653 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:26.872851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:26.932086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab1b974-11b5-401d-9873-f73df68af558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:26.932086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab1b974-11b5-401d-9873-f73df68af558 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:26.932086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab1b974-11b5-401d-9873-f73df68af558 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:26.932086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab1b974-11b5-401d-9873-f73df68af558 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:26.933033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab1b974-11b5-401d-9873-f73df68af558 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:26.934603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab1b974-11b5-401d-9873-f73df68af558 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:26.960260 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1095/2153] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:01:26 2022] POST /identity/v3/users => generated 292 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:01:26.961865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5159e3e-35cf-41b5-8507-992733cbe540 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:26.973334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5159e3e-35cf-41b5-8507-992733cbe540 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:26.982694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d97aec6-538f-4113-ba94-b69348c041a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:26.983335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d97aec6-538f-4113-ba94-b69348c041a0 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:26.983611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d97aec6-538f-4113-ba94-b69348c041a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:26.983932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d97aec6-538f-4113-ba94-b69348c041a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:26.984549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d97aec6-538f-4113-ba94-b69348c041a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:26.986134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d97aec6-538f-4113-ba94-b69348c041a0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:27.012177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1059/2154] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:26 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:27.016216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab1b974-11b5-401d-9873-f73df68af558 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:27.026485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab1b974-11b5-401d-9873-f73df68af558 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:27.031121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffb895a-58ff-4020-af78-5de26444e26e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:27.031715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb895a-58ff-4020-af78-5de26444e26e admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:27.031993 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb895a-58ff-4020-af78-5de26444e26e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:27.032308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb895a-58ff-4020-af78-5de26444e26e admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:27.032980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffb895a-58ff-4020-af78-5de26444e26e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:27.034558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffb895a-58ff-4020-af78-5de26444e26e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:27.046409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1096/2155] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:27.050933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d97aec6-538f-4113-ba94-b69348c041a0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:27.061677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d97aec6-538f-4113-ba94-b69348c041a0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:27.066286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:27.066937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:27.067217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:27.067493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] PATH_INFO: `/v3/projects/7e9c4dec37674f73912f0d9272ac2446/users/4f104bf97764426cb67b14bbf39e16be/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:27.083198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e9c4dec37674f73912f0d9272ac2446, user_id=4f104bf97764426cb67b14bbf39e16be, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:27.085874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:27.098679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1060/2156] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:01:27 2022] PUT /identity/v3/projects/7e9c4dec37674f73912f0d9272ac2446/users/4f104bf97764426cb67b14bbf39e16be/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:27.103758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a92abda-b3f4-4e72-a780-83ee6147dc3c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:27.104046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a92abda-b3f4-4e72-a780-83ee6147dc3c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:27.104680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a92abda-b3f4-4e72-a780-83ee6147dc3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:27.125018 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0a92abda-b3f4-4e72-a780-83ee6147dc3c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:27.126261 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0a92abda-b3f4-4e72-a780-83ee6147dc3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:27.187374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1097/2157] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:01:27.197040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:27.200034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:27.221040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:27.265006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51529f46-ba8f-4cd5-857a-8e04e20a5ff5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:27.269252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35b2d17-3ec3-49af-a962-9b8e312d0df5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:27.269860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35b2d17-3ec3-49af-a962-9b8e312d0df5 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:27.270141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35b2d17-3ec3-49af-a962-9b8e312d0df5 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:27.270417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35b2d17-3ec3-49af-a962-9b8e312d0df5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:27.283368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35b2d17-3ec3-49af-a962-9b8e312d0df5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:27.284839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35b2d17-3ec3-49af-a962-9b8e312d0df5 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:27.295190 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1061/2158] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:29.702193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a92abda-b3f4-4e72-a780-83ee6147dc3c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:29.703656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0a92abda-b3f4-4e72-a780-83ee6147dc3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:29.804879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a92abda-b3f4-4e72-a780-83ee6147dc3c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:29.811752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:29.813518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:29.813518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:29.813518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:29.815078 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:29.875589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:29.963444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:29.966375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:30.025340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1098/2159] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:29 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:31.784414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1a66a4-9301-42ab-830e-12e10a7e6965 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:31.784888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1a66a4-9301-42ab-830e-12e10a7e6965 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:31.785181 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1a66a4-9301-42ab-830e-12e10a7e6965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:31.829129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-ac1a66a4-9301-42ab-830e-12e10a7e6965 None None] MFA Rules not processed for user `4f104bf97764426cb67b14bbf39e16be`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:31.830703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ac1a66a4-9301-42ab-830e-12e10a7e6965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:31.952481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1062/2160] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:31.957122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:31.968543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff8063-5965-42f4-8b3c-5b86614b26d7 service nova] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:31.973594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb628d-4f55-495c-8b96-c66e07d7c12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:31.974203 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb628d-4f55-495c-8b96-c66e07d7c12d admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:31.974904 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb628d-4f55-495c-8b96-c66e07d7c12d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:31.975166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb628d-4f55-495c-8b96-c66e07d7c12d admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:31.975957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdb628d-4f55-495c-8b96-c66e07d7c12d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:31.977941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdb628d-4f55-495c-8b96-c66e07d7c12d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:32.030512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1099/2161] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:01:31 2022] POST /identity/v3/projects => generated 385 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:01:32.035492 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1a66a4-9301-42ab-830e-12e10a7e6965 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:32.050285 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1a66a4-9301-42ab-830e-12e10a7e6965 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:32.124884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3648b0c-0fb2-4823-94ec-485c269468f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:32.125650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3648b0c-0fb2-4823-94ec-485c269468f2 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:32.125934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3648b0c-0fb2-4823-94ec-485c269468f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:32.126209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3648b0c-0fb2-4823-94ec-485c269468f2 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:32.127194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3648b0c-0fb2-4823-94ec-485c269468f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:32.129557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3648b0c-0fb2-4823-94ec-485c269468f2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:32.159925 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1063/2162] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:01:32 2022] POST /identity/v3/users => generated 299 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:01:32.164039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb628d-4f55-495c-8b96-c66e07d7c12d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:32.175642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb628d-4f55-495c-8b96-c66e07d7c12d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:32.182025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8af366-a0cc-4f28-9335-41e07beb2803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:32.183548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8af366-a0cc-4f28-9335-41e07beb2803 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:32.183834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8af366-a0cc-4f28-9335-41e07beb2803 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:32.184103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8af366-a0cc-4f28-9335-41e07beb2803 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:32.184843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8af366-a0cc-4f28-9335-41e07beb2803 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:32.186477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8af366-a0cc-4f28-9335-41e07beb2803 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:32.201239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1100/2163] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:32.205224 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3648b0c-0fb2-4823-94ec-485c269468f2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:32.216451 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3648b0c-0fb2-4823-94ec-485c269468f2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:32.222250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ead3e0-c4ff-45ab-9e1a-bb8ed6b66c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:32.222993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ead3e0-c4ff-45ab-9e1a-bb8ed6b66c26 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:32.223341 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ead3e0-c4ff-45ab-9e1a-bb8ed6b66c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:32.223692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ead3e0-c4ff-45ab-9e1a-bb8ed6b66c26 admin admin] PATH_INFO: `/v3/projects/5f511c1474ea4a0787b6c7dfd68c16c9/users/f8616ecca9c543909db87c0ffcf511bc/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:32.241926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ead3e0-c4ff-45ab-9e1a-bb8ed6b66c26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f511c1474ea4a0787b6c7dfd68c16c9, user_id=f8616ecca9c543909db87c0ffcf511bc, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:32.245448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ead3e0-c4ff-45ab-9e1a-bb8ed6b66c26 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:32.265551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1064/2164] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:01:32 2022] PUT /identity/v3/projects/5f511c1474ea4a0787b6c7dfd68c16c9/users/f8616ecca9c543909db87c0ffcf511bc/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:32.266503 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8af366-a0cc-4f28-9335-41e07beb2803 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:32.280573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8af366-a0cc-4f28-9335-41e07beb2803 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:32.349413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09716cd-ff46-4079-ae58-f9f889e77438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:32.350077 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09716cd-ff46-4079-ae58-f9f889e77438 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:32.350358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09716cd-ff46-4079-ae58-f9f889e77438 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:32.350780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09716cd-ff46-4079-ae58-f9f889e77438 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:32.351423 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09716cd-ff46-4079-ae58-f9f889e77438 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:32.353111 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09716cd-ff46-4079-ae58-f9f889e77438 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:32.368793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1101/2165] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:32.371766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ead3e0-c4ff-45ab-9e1a-bb8ed6b66c26 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:32.390834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ead3e0-c4ff-45ab-9e1a-bb8ed6b66c26 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:32.396480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256909ad-f929-44ea-bce0-788eb07aeccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:32.397125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256909ad-f929-44ea-bce0-788eb07aeccf admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:32.397406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256909ad-f929-44ea-bce0-788eb07aeccf admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:32.397682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256909ad-f929-44ea-bce0-788eb07aeccf admin admin] PATH_INFO: `/v3/projects/5f511c1474ea4a0787b6c7dfd68c16c9/users/f8616ecca9c543909db87c0ffcf511bc/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:32.399748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256909ad-f929-44ea-bce0-788eb07aeccf admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f511c1474ea4a0787b6c7dfd68c16c9, user_id=f8616ecca9c543909db87c0ffcf511bc, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:32.402920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256909ad-f929-44ea-bce0-788eb07aeccf admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:32.416184 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1065/2166] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:01:32 2022] PUT /identity/v3/projects/5f511c1474ea4a0787b6c7dfd68c16c9/users/f8616ecca9c543909db87c0ffcf511bc/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:36.509000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09716cd-ff46-4079-ae58-f9f889e77438 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:36.637794 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:36.672116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09716cd-ff46-4079-ae58-f9f889e77438 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:36.834164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:36.834926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:36.836170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:36.836170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:36.849603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:36.873838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:36.876540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:36.893223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1102/2167] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:38.052962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4a9a08-dd65-4d80-acaf-cfa3a1928e9c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:38.053436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4a9a08-dd65-4d80-acaf-cfa3a1928e9c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:38.053436 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4a9a08-dd65-4d80-acaf-cfa3a1928e9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:38.061379 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:38.084897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-ae4a9a08-dd65-4d80-acaf-cfa3a1928e9c None None] MFA Rules not processed for user `f8616ecca9c543909db87c0ffcf511bc`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:38.086121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ae4a9a08-dd65-4d80-acaf-cfa3a1928e9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:38.165778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1066/2168] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:38 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:38.184394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:38.186496 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:38.270063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d74651-1ee4-4bc5-a0f4-028d9a9ffeeb service nova] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:38.274015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a4fd11d5d85479c9aa1d87b23a13141', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:38.274838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:38.275117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:38.275394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:38.343479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:38.344901 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:38.354266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1103/2169] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:38.512004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584f532-bb75-421d-80e5-70b6cfe693ca None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:38.512625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584f532-bb75-421d-80e5-70b6cfe693ca None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:38.513119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584f532-bb75-421d-80e5-70b6cfe693ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:38.580343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c584f532-bb75-421d-80e5-70b6cfe693ca None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:38.581583 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c584f532-bb75-421d-80e5-70b6cfe693ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:38.700512 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1067/2170] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 09:01:38 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:38.790129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:38.791894 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:38.819105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:38.943014 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b3287f-0c9b-4e24-b96f-d14c41792d2f service glance] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:38.948571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:38.949224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:38.949506 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:38.949786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:38.972461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:38.974467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:38.986168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1104/2171] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 09:01:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 09:01:43.403816 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c584f532-bb75-421d-80e5-70b6cfe693ca None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:43.405380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c584f532-bb75-421d-80e5-70b6cfe693ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:43.475934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c584f532-bb75-421d-80e5-70b6cfe693ca None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:43.480553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:43.481452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:43.481896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:43.482354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:43.485214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:43.504750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:43.571743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:43.573491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:43.586709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1068/2172] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:47.483224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:47.484804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:47.648000 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12618a1-1778-4ef3-8737-463f84e764c2 service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:47.656313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:47.657300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:47.658131 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:47.660563 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:47.664112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:47.806581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:47.809385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:47.814641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:47.870666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1105/2173] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:47 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:49.513817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:49.515332 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:49.529174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e7da54-cf61-412f-b89f-441894d212a0 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:49.534447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:49.535330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:49.535799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:49.536439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] PATH_INFO: `/v3/users/94169e17266b4e6fad7e068a4c1384c7` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:49.538170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] RBAC: Authorizing `identity:delete_user(user_id=94169e17266b4e6fad7e068a4c1384c7)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:49.540426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:49.623719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94169e17266b4e6fad7e068a4c1384c7'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:49.633000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94169e17266b4e6fad7e068a4c1384c7'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:49.653546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94169e17266b4e6fad7e068a4c1384c7'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:49.654380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94169e17266b4e6fad7e068a4c1384c7'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:49.655047 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-075034aa-cc51-4ae2-83fe-fe5bc5b41efd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94169e17266b4e6fad7e068a4c1384c7'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:49.679677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1069/2174] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:01:49 2022] DELETE /identity/v3/users/94169e17266b4e6fad7e068a4c1384c7 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:50.166178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:50.186679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b356bb65-57dc-41d2-899b-78ecf9ff4fd4 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:50.318600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:50.319936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:50.320271 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:50.320550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] PATH_INFO: `/v3/projects/45be9d45ba4c4a2ca1ed053a7263dbee` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:50.321749 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=45be9d45ba4c4a2ca1ed053a7263dbee)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:50.324555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:50.387991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45be9d45ba4c4a2ca1ed053a7263dbee', 'request_id': 'req-6af6f27b-8577-45bb-8c9b-2541d175ce1e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:50.394857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45be9d45ba4c4a2ca1ed053a7263dbee', 'request_id': 'req-6af6f27b-8577-45bb-8c9b-2541d175ce1e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:50.404385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45be9d45ba4c4a2ca1ed053a7263dbee', 'request_id': 'req-6af6f27b-8577-45bb-8c9b-2541d175ce1e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:50.419713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] The token cache is being invalidate because project 45be9d45ba4c4a2ca1ed053a7263dbee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:01:50.420009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:50.420751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-6af6f27b-8577-45bb-8c9b-2541d175ce1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:50.428556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae50bc0c-923d-4ee2-8ade-03ead862823f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:50.429200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae50bc0c-923d-4ee2-8ade-03ead862823f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:50.429200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae50bc0c-923d-4ee2-8ade-03ead862823f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:50.432465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1106/2175] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:01:50 2022] DELETE /identity/v3/projects/45be9d45ba4c4a2ca1ed053a7263dbee => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:50.454576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-ae50bc0c-923d-4ee2-8ade-03ead862823f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:50.455793 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ae50bc0c-923d-4ee2-8ade-03ead862823f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:50.535589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1070/2176] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:50.545060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437dfed4-2395-40c1-8321-5330dafb5df9 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:50.545380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437dfed4-2395-40c1-8321-5330dafb5df9 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:50.545692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437dfed4-2395-40c1-8321-5330dafb5df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:50.574592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-437dfed4-2395-40c1-8321-5330dafb5df9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:50.576226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-437dfed4-2395-40c1-8321-5330dafb5df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:50.589426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1107/2177] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:01:50.590933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae50bc0c-923d-4ee2-8ade-03ead862823f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:50.606640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae50bc0c-923d-4ee2-8ade-03ead862823f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:50.612609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:50.613568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:50.614538 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:50.615383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:50.616886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:01:50.617543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:50.620361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:50.641359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1071/2178] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:01:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 09:01:50.648279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437dfed4-2395-40c1-8321-5330dafb5df9 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:50.661743 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437dfed4-2395-40c1-8321-5330dafb5df9 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:50.675639 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657da59-4601-4b06-a5dd-4de19a802629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:50.677516 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657da59-4601-4b06-a5dd-4de19a802629 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:50.677972 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657da59-4601-4b06-a5dd-4de19a802629 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:50.678407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657da59-4601-4b06-a5dd-4de19a802629 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:50.679369 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e657da59-4601-4b06-a5dd-4de19a802629 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:50.682276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e657da59-4601-4b06-a5dd-4de19a802629 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:50.721803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1108/2179] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:01:50 2022] POST /identity/v3/projects => generated 377 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:01:50.726357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:50.743550 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a1bed7-0ea1-46f5-a41f-1ea54b32e2b0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:50.753129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:50.860421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc19a68-fbe0-4c9d-81f9-806693bd48f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:50.861178 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc19a68-fbe0-4c9d-81f9-806693bd48f1 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:50.861581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc19a68-fbe0-4c9d-81f9-806693bd48f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:50.861920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc19a68-fbe0-4c9d-81f9-806693bd48f1 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:50.862649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc19a68-fbe0-4c9d-81f9-806693bd48f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:50.865083 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc19a68-fbe0-4c9d-81f9-806693bd48f1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:50.895593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1072/2180] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:01:50 2022] POST /identity/v3/users => generated 289 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:01:50.900543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657da59-4601-4b06-a5dd-4de19a802629 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:50.915704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657da59-4601-4b06-a5dd-4de19a802629 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:50.920802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f7e5ed-b8c5-4dd7-9786-3da21b7cda4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:50.921413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f7e5ed-b8c5-4dd7-9786-3da21b7cda4c admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:50.921691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f7e5ed-b8c5-4dd7-9786-3da21b7cda4c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:50.922103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f7e5ed-b8c5-4dd7-9786-3da21b7cda4c admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:50.922754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f7e5ed-b8c5-4dd7-9786-3da21b7cda4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:50.924364 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f7e5ed-b8c5-4dd7-9786-3da21b7cda4c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:50.978110 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1109/2181] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:50 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:50.990617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc19a68-fbe0-4c9d-81f9-806693bd48f1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:51.009289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc19a68-fbe0-4c9d-81f9-806693bd48f1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:51.021194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f3b940-fe16-424b-b05c-fdf43303efa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:51.022727 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f3b940-fe16-424b-b05c-fdf43303efa9 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:51.023353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f3b940-fe16-424b-b05c-fdf43303efa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:51.024094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f3b940-fe16-424b-b05c-fdf43303efa9 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:51.025054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f3b940-fe16-424b-b05c-fdf43303efa9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:51.027581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f3b940-fe16-424b-b05c-fdf43303efa9 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:51.064240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1073/2182] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:50 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:51.068272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f7e5ed-b8c5-4dd7-9786-3da21b7cda4c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:51.088012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f7e5ed-b8c5-4dd7-9786-3da21b7cda4c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:51.095090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78377512-83a3-4047-ad97-a8afc09dbe04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:51.095756 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78377512-83a3-4047-ad97-a8afc09dbe04 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:51.096216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78377512-83a3-4047-ad97-a8afc09dbe04 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:51.096470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78377512-83a3-4047-ad97-a8afc09dbe04 admin admin] PATH_INFO: `/v3/projects/057bb2c50f1a40e18f7cd9520f05837a/users/bd7d49cdf33d403fb898ed55108485c5/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:51.117147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78377512-83a3-4047-ad97-a8afc09dbe04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=057bb2c50f1a40e18f7cd9520f05837a, user_id=bd7d49cdf33d403fb898ed55108485c5, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:51.127714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78377512-83a3-4047-ad97-a8afc09dbe04 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:51.153065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1110/2183] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:01:51 2022] PUT /identity/v3/projects/057bb2c50f1a40e18f7cd9520f05837a/users/bd7d49cdf33d403fb898ed55108485c5/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:51.164525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a99978-e671-4116-9e7f-70da07c57a8d None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:51.164949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a99978-e671-4116-9e7f-70da07c57a8d None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:51.165409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a99978-e671-4116-9e7f-70da07c57a8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:51.247442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f4a99978-e671-4116-9e7f-70da07c57a8d None None] MFA Rules not processed for user `bd7d49cdf33d403fb898ed55108485c5`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:51.248865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f4a99978-e671-4116-9e7f-70da07c57a8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:51.442239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1074/2184] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:51 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:51.448965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78377512-83a3-4047-ad97-a8afc09dbe04 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:51.474591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:51.571068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78377512-83a3-4047-ad97-a8afc09dbe04 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:51.579204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30afe27c-6857-417d-b72d-232aa44676b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:51.580768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30afe27c-6857-417d-b72d-232aa44676b1 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:51.581340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30afe27c-6857-417d-b72d-232aa44676b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:51.581983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30afe27c-6857-417d-b72d-232aa44676b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:51.583758 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30afe27c-6857-417d-b72d-232aa44676b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:51.586982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30afe27c-6857-417d-b72d-232aa44676b1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:51.629731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1111/2185] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:01:51 2022] POST /identity/v3/projects => generated 379 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:01:51.630971 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a99978-e671-4116-9e7f-70da07c57a8d None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:51.642449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a99978-e671-4116-9e7f-70da07c57a8d None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:51.730344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4044b255-fc11-4aa5-b073-a798833ef41f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:51.731093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4044b255-fc11-4aa5-b073-a798833ef41f admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:51.731831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4044b255-fc11-4aa5-b073-a798833ef41f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:51.732389 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4044b255-fc11-4aa5-b073-a798833ef41f admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:51.733233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4044b255-fc11-4aa5-b073-a798833ef41f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:51.735574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4044b255-fc11-4aa5-b073-a798833ef41f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:51.761290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1075/2186] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:01:51 2022] POST /identity/v3/users => generated 296 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:01:51.766134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30afe27c-6857-417d-b72d-232aa44676b1 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:51.778204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30afe27c-6857-417d-b72d-232aa44676b1 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:51.787171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc9f13-c7fe-4c08-9c48-72a5946cb238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:51.788075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc9f13-c7fe-4c08-9c48-72a5946cb238 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:51.788380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc9f13-c7fe-4c08-9c48-72a5946cb238 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:51.788658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc9f13-c7fe-4c08-9c48-72a5946cb238 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:51.789306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fc9f13-c7fe-4c08-9c48-72a5946cb238 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:51.790922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fc9f13-c7fe-4c08-9c48-72a5946cb238 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:51.813361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1112/2187] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:51.817817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4044b255-fc11-4aa5-b073-a798833ef41f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:51.828551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4044b255-fc11-4aa5-b073-a798833ef41f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:51.840684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3875f4-8296-46b2-bd37-574e9e6c925e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:51.841909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3875f4-8296-46b2-bd37-574e9e6c925e admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:51.842369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3875f4-8296-46b2-bd37-574e9e6c925e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:51.842818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3875f4-8296-46b2-bd37-574e9e6c925e admin admin] PATH_INFO: `/v3/projects/200d8995e348488097bd229b7323ac9e/users/0bd7d94f71cf4b848d51e3c23bd917ae/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:51.847049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:51.869534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3875f4-8296-46b2-bd37-574e9e6c925e admin admin] RBAC: Authorizing `identity:create_grant(project_id=200d8995e348488097bd229b7323ac9e, user_id=0bd7d94f71cf4b848d51e3c23bd917ae, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:51.872566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3875f4-8296-46b2-bd37-574e9e6c925e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:51.893517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc9f13-c7fe-4c08-9c48-72a5946cb238 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:51.894974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1076/2188] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:01:51 2022] PUT /identity/v3/projects/200d8995e348488097bd229b7323ac9e/users/0bd7d94f71cf4b848d51e3c23bd917ae/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:01:51.905394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc9f13-c7fe-4c08-9c48-72a5946cb238 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:51.983273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3903991-fc10-4621-8889-c95a8a696222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:51.986319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3903991-fc10-4621-8889-c95a8a696222 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:51.988030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3903991-fc10-4621-8889-c95a8a696222 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:51.991339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3903991-fc10-4621-8889-c95a8a696222 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:51.992250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3903991-fc10-4621-8889-c95a8a696222 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:51.996398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3903991-fc10-4621-8889-c95a8a696222 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:52.009586 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1113/2189] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:52.014155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3875f4-8296-46b2-bd37-574e9e6c925e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:52.034847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3875f4-8296-46b2-bd37-574e9e6c925e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:52.040828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a1ecfb-d3f4-42f9-b54b-83152d2f5ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:52.041646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a1ecfb-d3f4-42f9-b54b-83152d2f5ae6 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:52.042774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a1ecfb-d3f4-42f9-b54b-83152d2f5ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:52.043141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a1ecfb-d3f4-42f9-b54b-83152d2f5ae6 admin admin] PATH_INFO: `/v3/projects/200d8995e348488097bd229b7323ac9e/users/0bd7d94f71cf4b848d51e3c23bd917ae/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:52.045259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a1ecfb-d3f4-42f9-b54b-83152d2f5ae6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=200d8995e348488097bd229b7323ac9e, user_id=0bd7d94f71cf4b848d51e3c23bd917ae, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:52.048050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a1ecfb-d3f4-42f9-b54b-83152d2f5ae6 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:52.068716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1077/2190] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:01:52 2022] PUT /identity/v3/projects/200d8995e348488097bd229b7323ac9e/users/0bd7d94f71cf4b848d51e3c23bd917ae/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:01:52.072662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f730461c-ab63-4ce7-b2cc-611176cfab69 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:52.073052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f730461c-ab63-4ce7-b2cc-611176cfab69 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:52.073448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f730461c-ab63-4ce7-b2cc-611176cfab69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:52.099256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1bb772-1edb-4107-82f9-7e5b3f48374f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:52.099560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1bb772-1edb-4107-82f9-7e5b3f48374f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:52.099815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1bb772-1edb-4107-82f9-7e5b3f48374f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:52.114977 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-f730461c-ab63-4ce7-b2cc-611176cfab69 None None] MFA Rules not processed for user `0bd7d94f71cf4b848d51e3c23bd917ae`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:52.117390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f730461c-ab63-4ce7-b2cc-611176cfab69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:52.121578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5d1bb772-1edb-4107-82f9-7e5b3f48374f None None] MFA Rules not processed for user `4f104bf97764426cb67b14bbf39e16be`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:52.122805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5d1bb772-1edb-4107-82f9-7e5b3f48374f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:52.188931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1078/2191] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:01:52.203022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1114/2192] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:52 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:52.220282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1bb772-1edb-4107-82f9-7e5b3f48374f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:52.222881 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5d1bb772-1edb-4107-82f9-7e5b3f48374f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:52.285415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1bb772-1edb-4107-82f9-7e5b3f48374f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:52.289101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b30d11-784e-44ce-ad5b-3007a6b728e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:52.289778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b30d11-784e-44ce-ad5b-3007a6b728e0 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:52.290124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b30d11-784e-44ce-ad5b-3007a6b728e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:52.290456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b30d11-784e-44ce-ad5b-3007a6b728e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:52.302380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b30d11-784e-44ce-ad5b-3007a6b728e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:52.303955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b30d11-784e-44ce-ad5b-3007a6b728e0 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:52.316018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1079/2193] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:52 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:52.535737 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f730461c-ab63-4ce7-b2cc-611176cfab69 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:52.537673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f730461c-ab63-4ce7-b2cc-611176cfab69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:52.558182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:52.633087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f730461c-ab63-4ce7-b2cc-611176cfab69 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:52.638389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:52.639057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:52.639519 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:52.639949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:52.642804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:52.656560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:52.658122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:52.677085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1115/2194] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:52.781340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b30d11-784e-44ce-ad5b-3007a6b728e0 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:52.801013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b30d11-784e-44ce-ad5b-3007a6b728e0 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:52.806876 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bd7d94f71cf4b848d51e3c23bd917ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '200d8995e348488097bd229b7323ac9e', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:52.807825 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:52.808396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:52.808924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:52.810456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:01:52.811450 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:52.813773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:52.833180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1080/2195] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:01:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:01:52.837499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:52.856375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff40957-9c82-4d78-b65b-40cc11c50c15 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:52.862957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2663a6c-683b-446b-8408-58bb824fdcb8 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bd7d94f71cf4b848d51e3c23bd917ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '200d8995e348488097bd229b7323ac9e', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:52.863564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2663a6c-683b-446b-8408-58bb824fdcb8 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:52.863850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2663a6c-683b-446b-8408-58bb824fdcb8 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:52.864126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2663a6c-683b-446b-8408-58bb824fdcb8 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:52.865131 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2663a6c-683b-446b-8408-58bb824fdcb8 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:52.867253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2663a6c-683b-446b-8408-58bb824fdcb8 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:52.903192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1116/2196] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:01:52 2022] POST /identity/v3/projects => generated 354 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:01:52.907344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:52.918969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a1164a-dfe1-41c1-b8fa-b105325ccb80 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:52.995703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bd7d94f71cf4b848d51e3c23bd917ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '200d8995e348488097bd229b7323ac9e', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:52.996385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:52.996677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:52.997034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:52.997576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:01:52.997886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:52.999581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:53.008845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:53.019339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1081/2197] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:01:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:01:53.063317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2663a6c-683b-446b-8408-58bb824fdcb8 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:53.079180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2663a6c-683b-446b-8408-58bb824fdcb8 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:53.083506 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bd7d94f71cf4b848d51e3c23bd917ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '200d8995e348488097bd229b7323ac9e', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:53.084189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:53.084539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:53.084869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] PATH_INFO: `/v3/projects/506d363ce35341c7b0fa98da680d8a2d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:53.086123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Authorizing `identity:delete_project(project_id=506d363ce35341c7b0fa98da680d8a2d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:53.088081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:53.109685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:53.112621 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:53.166184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '506d363ce35341c7b0fa98da680d8a2d', 'request_id': 'req-232d36c1-3b4e-472f-9117-e26c89024d66'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.174589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '506d363ce35341c7b0fa98da680d8a2d', 'request_id': 'req-232d36c1-3b4e-472f-9117-e26c89024d66'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.185695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '506d363ce35341c7b0fa98da680d8a2d', 'request_id': 'req-232d36c1-3b4e-472f-9117-e26c89024d66'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.199398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] The token cache is being invalidate because project 506d363ce35341c7b0fa98da680d8a2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:01:53.199710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.200396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.206573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1117/2198] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:01:53 2022] DELETE /identity/v3/projects/506d363ce35341c7b0fa98da680d8a2d => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:53.217235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810cac04-5b37-415d-be4e-0de2e92b9603 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:53.294860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232b07f5-357d-47b3-b587-0f1020e4f6c5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:53.295518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b07f5-357d-47b3-b587-0f1020e4f6c5 service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:53.295884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b07f5-357d-47b3-b587-0f1020e4f6c5 service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:53.296193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b07f5-357d-47b3-b587-0f1020e4f6c5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:53.298120 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-232b07f5-357d-47b3-b587-0f1020e4f6c5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:53.311277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232b07f5-357d-47b3-b587-0f1020e4f6c5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:53.313306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232b07f5-357d-47b3-b587-0f1020e4f6c5 service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:53.325369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1082/2199] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:53.456451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:53.458125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:53.548172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232d36c1-3b4e-472f-9117-e26c89024d66 tempest-VolumeQuotaClassesTest-423449182 tempest-VolumeQuotaClassesTest-423449182-project-admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:53.554456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:53.555100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:53.555398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:53.555662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] PATH_INFO: `/v3/users/bd7d49cdf33d403fb898ed55108485c5` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:53.556819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd7d49cdf33d403fb898ed55108485c5)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:53.558780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:53.582980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:53.612295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd7d49cdf33d403fb898ed55108485c5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.620595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd7d49cdf33d403fb898ed55108485c5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.633382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd7d49cdf33d403fb898ed55108485c5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.634239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd7d49cdf33d403fb898ed55108485c5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.634802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd7d49cdf33d403fb898ed55108485c5'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:53.649134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1118/2200] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:01:53 2022] DELETE /identity/v3/users/bd7d49cdf33d403fb898ed55108485c5 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:53.655233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da21861-fe37-4f4d-82b4-fe6821185a9a None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:53.655540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da21861-fe37-4f4d-82b4-fe6821185a9a None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:53.655819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da21861-fe37-4f4d-82b4-fe6821185a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:53.675624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2da21861-fe37-4f4d-82b4-fe6821185a9a None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:53.676877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2da21861-fe37-4f4d-82b4-fe6821185a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:53.756813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1083/2201] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:53.767471 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:53.768932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:53.832177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92aa742-58ff-4c2f-9a6d-5eed2372e24d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:53.836075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:53.836883 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:53.837167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:53.837440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:53.849498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:53.853484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:53.863295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1119/2202] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:53.946019 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da21861-fe37-4f4d-82b4-fe6821185a9a None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:53.959885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da21861-fe37-4f4d-82b4-fe6821185a9a None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:53.965194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:53.966024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:53.966693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:53.967283 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:53.969386 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:53.995745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:53.997232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:54.012483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1084/2203] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:54.262107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:54.263917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:54.279016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc3bf55-7442-4759-b819-767419956c70 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:54.284535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:54.285430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:54.285430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:54.285885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] PATH_INFO: `/v3/projects/057bb2c50f1a40e18f7cd9520f05837a` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:54.287926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=057bb2c50f1a40e18f7cd9520f05837a)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:54.289844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:54.347295 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '057bb2c50f1a40e18f7cd9520f05837a', 'request_id': 'req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.354040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '057bb2c50f1a40e18f7cd9520f05837a', 'request_id': 'req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.362698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '057bb2c50f1a40e18f7cd9520f05837a', 'request_id': 'req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.374424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] The token cache is being invalidate because project 057bb2c50f1a40e18f7cd9520f05837a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:01:54.374718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.375419 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ca9475ef-6250-469b-8d0d-3e1f7c49b5a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.376953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1120/2204] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:01:54 2022] DELETE /identity/v3/projects/057bb2c50f1a40e18f7cd9520f05837a => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:54.381013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:54.391136 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190eed29-8163-4919-bff0-433c360b6898 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:54.404648 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:54.461304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:54.461956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:54.462236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:54.462649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] PATH_INFO: `/v3/users/0bd7d94f71cf4b848d51e3c23bd917ae` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:54.465811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0bd7d94f71cf4b848d51e3c23bd917ae)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:54.469278 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:54.553325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0bd7d94f71cf4b848d51e3c23bd917ae'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.576835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0bd7d94f71cf4b848d51e3c23bd917ae'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.614660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0bd7d94f71cf4b848d51e3c23bd917ae'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.616195 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0bd7d94f71cf4b848d51e3c23bd917ae'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.618326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0bd7d94f71cf4b848d51e3c23bd917ae'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:54.645495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1085/2205] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:01:54 2022] DELETE /identity/v3/users/0bd7d94f71cf4b848d51e3c23bd917ae => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:54.917730 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2fd5e2-c241-49a9-b7b1-e7381f4393ae None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:54.918497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2fd5e2-c241-49a9-b7b1-e7381f4393ae None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:54.918497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2fd5e2-c241-49a9-b7b1-e7381f4393ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:54.941586 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4c2fd5e2-c241-49a9-b7b1-e7381f4393ae None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:54.943109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4c2fd5e2-c241-49a9-b7b1-e7381f4393ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:54.962150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:55.019253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1121/2206] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 09:01:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:01:55.034477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:55.036612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:55.127862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2847322-f6f2-4dff-bcda-b18950d6dc68 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:55.134516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1347ece3-7fb9-4be2-b320-058b551bf8ed service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:55.134646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1347ece3-7fb9-4be2-b320-058b551bf8ed service swift] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:55.134930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1347ece3-7fb9-4be2-b320-058b551bf8ed service swift] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:55.135201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1347ece3-7fb9-4be2-b320-058b551bf8ed service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:55.147893 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1347ece3-7fb9-4be2-b320-058b551bf8ed service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:55.149795 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1347ece3-7fb9-4be2-b320-058b551bf8ed service swift] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:55.160043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1086/2207] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 09:01:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 09:01:55.456118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2fd5e2-c241-49a9-b7b1-e7381f4393ae None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:55.458238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4c2fd5e2-c241-49a9-b7b1-e7381f4393ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:55.707855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2fd5e2-c241-49a9-b7b1-e7381f4393ae None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:55.730683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:55.731159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:55.731443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:55.731715 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] PATH_INFO: `/v3/projects/200d8995e348488097bd229b7323ac9e` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:55.744185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] RBAC: Authorizing `identity:delete_project(project_id=200d8995e348488097bd229b7323ac9e)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:55.746347 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:55.897085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '200d8995e348488097bd229b7323ac9e', 'request_id': 'req-42e0be98-5782-482d-a7d3-363249f0e04b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:55.915675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '200d8995e348488097bd229b7323ac9e', 'request_id': 'req-42e0be98-5782-482d-a7d3-363249f0e04b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:55.931171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '200d8995e348488097bd229b7323ac9e', 'request_id': 'req-42e0be98-5782-482d-a7d3-363249f0e04b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:55.960256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] The token cache is being invalidate because project 200d8995e348488097bd229b7323ac9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:01:55.961610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:55.963714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:55.970113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1122/2208] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:01:55 2022] DELETE /identity/v3/projects/200d8995e348488097bd229b7323ac9e => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:55.979033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bb5ebe-f38d-4b56-9846-3850501f681e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:55.979338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bb5ebe-f38d-4b56-9846-3850501f681e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:55.980245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bb5ebe-f38d-4b56-9846-3850501f681e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:56.028528 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-78bb5ebe-f38d-4b56-9846-3850501f681e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:56.029931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-78bb5ebe-f38d-4b56-9846-3850501f681e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:56.077979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:56.203396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1087/2209] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:56.207312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:56.212866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:56.219226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6b27-1b8e-4522-a031-46cc1a927e75 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:56.230611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6b27-1b8e-4522-a031-46cc1a927e75 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:56.231662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6b27-1b8e-4522-a031-46cc1a927e75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:56.288950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3a8a6b27-1b8e-4522-a031-46cc1a927e75 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:56.293033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3a8a6b27-1b8e-4522-a031-46cc1a927e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:56.315610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1088/2210] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:56.318177 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a6b27-1b8e-4522-a031-46cc1a927e75 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:56.338743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a6b27-1b8e-4522-a031-46cc1a927e75 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:56.355247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:56.357979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:56.359337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:56.359846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:56.363088 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:01:56.363748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:56.368611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:56.378020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e0be98-5782-482d-a7d3-363249f0e04b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:56.390141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:56.393363 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:56.394199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:56.394531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:56.399714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:56.407056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1089/2211] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:01:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:01:56.408668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:56.432442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f2bd4b-c0c4-4c27-98ba-7bc2875fba92 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:56.437548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:56.441063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:56.447149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f2c35-80ec-4e36-91f3-f01df7ed9a9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:56.447776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f2c35-80ec-4e36-91f3-f01df7ed9a9e admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:56.448089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f2c35-80ec-4e36-91f3-f01df7ed9a9e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:56.448476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f2c35-80ec-4e36-91f3-f01df7ed9a9e admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:56.449702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1f2c35-80ec-4e36-91f3-f01df7ed9a9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:56.456225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:56.457142 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1f2c35-80ec-4e36-91f3-f01df7ed9a9e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:56.468952 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1123/2212] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:01:56.548956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1090/2213] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:01:56 2022] POST /identity/v3/projects => generated 361 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:01:56.551434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:56.571885 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c3753d-79be-4f28-831c-996842889462 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:56.769397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d95096-6494-4af4-a360-c732fb7690e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:56.773032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d95096-6494-4af4-a360-c732fb7690e6 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:56.773480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d95096-6494-4af4-a360-c732fb7690e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:56.774331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d95096-6494-4af4-a360-c732fb7690e6 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:56.775861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d95096-6494-4af4-a360-c732fb7690e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:56.780490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d95096-6494-4af4-a360-c732fb7690e6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:56.832340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1124/2214] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:01:56 2022] POST /identity/v3/users => generated 281 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:01:56.833703 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f2c35-80ec-4e36-91f3-f01df7ed9a9e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:56.850518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f2c35-80ec-4e36-91f3-f01df7ed9a9e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:56.863974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0788ef67-a01d-4c6b-9025-c3d39b8a1c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:56.866502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0788ef67-a01d-4c6b-9025-c3d39b8a1c9a admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:56.869531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0788ef67-a01d-4c6b-9025-c3d39b8a1c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:56.870016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0788ef67-a01d-4c6b-9025-c3d39b8a1c9a admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:56.872546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0788ef67-a01d-4c6b-9025-c3d39b8a1c9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:56.874356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0788ef67-a01d-4c6b-9025-c3d39b8a1c9a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:56.904753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1091/2215] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:56.906179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d95096-6494-4af4-a360-c732fb7690e6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:56.920887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d95096-6494-4af4-a360-c732fb7690e6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:56.931841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce196d8-5b91-405c-af85-3b5f55452b73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:56.933752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce196d8-5b91-405c-af85-3b5f55452b73 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:56.933752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce196d8-5b91-405c-af85-3b5f55452b73 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:56.937664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce196d8-5b91-405c-af85-3b5f55452b73 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:56.941903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce196d8-5b91-405c-af85-3b5f55452b73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:56.950104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce196d8-5b91-405c-af85-3b5f55452b73 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:57.014992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1125/2216] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:01:57.015653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0788ef67-a01d-4c6b-9025-c3d39b8a1c9a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:57.033651 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0788ef67-a01d-4c6b-9025-c3d39b8a1c9a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:57.054737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:57.055619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:57.058887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:57.061135 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] PATH_INFO: `/v3/projects/33e1e7c36252488aa8a2255b3c44c498/users/5aab6e26af8944c483ada8f20e3a2dbe/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:57.126580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33e1e7c36252488aa8a2255b3c44c498, user_id=5aab6e26af8944c483ada8f20e3a2dbe, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:57.129393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:57.169008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1092/2217] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:01:57 2022] PUT /identity/v3/projects/33e1e7c36252488aa8a2255b3c44c498/users/5aab6e26af8944c483ada8f20e3a2dbe/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:57.170073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7513ad8-0121-47bf-815b-c00ad08e23fc None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:57.170504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7513ad8-0121-47bf-815b-c00ad08e23fc None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:57.170876 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7513ad8-0121-47bf-815b-c00ad08e23fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:57.239510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d7513ad8-0121-47bf-815b-c00ad08e23fc None None] MFA Rules not processed for user `5aab6e26af8944c483ada8f20e3a2dbe`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:57.241585 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d7513ad8-0121-47bf-815b-c00ad08e23fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:57.434310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1126/2218] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:57 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:57.472587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:57.476734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:57.626044 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6b4603-3d1b-4e92-8323-4b036bb1a057 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:57.635317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1db27b-2e6a-49d3-8f59-13da0e283e2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:57.638091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1db27b-2e6a-49d3-8f59-13da0e283e2e service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:57.638344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1db27b-2e6a-49d3-8f59-13da0e283e2e service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:57.638628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1db27b-2e6a-49d3-8f59-13da0e283e2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:57.690033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1db27b-2e6a-49d3-8f59-13da0e283e2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:57.691512 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1db27b-2e6a-49d3-8f59-13da0e283e2e service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:57.712430 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1093/2219] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:57 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:57.729451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7513ad8-0121-47bf-815b-c00ad08e23fc None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:57.868782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7513ad8-0121-47bf-815b-c00ad08e23fc None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:57.885556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:57.887149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:57.891214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:57.891978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] PATH_INFO: `/v3/users/5aab6e26af8944c483ada8f20e3a2dbe` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:57.894846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5aab6e26af8944c483ada8f20e3a2dbe)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:57.902660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:57.959976 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:58.047773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5aab6e26af8944c483ada8f20e3a2dbe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:58.069472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5aab6e26af8944c483ada8f20e3a2dbe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:58.112361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5aab6e26af8944c483ada8f20e3a2dbe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:58.116663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5aab6e26af8944c483ada8f20e3a2dbe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:58.121856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5aab6e26af8944c483ada8f20e3a2dbe'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:58.163356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1127/2220] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:01:57 2022] DELETE /identity/v3/users/5aab6e26af8944c483ada8f20e3a2dbe => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:58.168435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab8e453-faae-4064-84eb-82bced1a6890 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:58.169578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab8e453-faae-4064-84eb-82bced1a6890 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:58.169874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab8e453-faae-4064-84eb-82bced1a6890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:58.203455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-3ab8e453-faae-4064-84eb-82bced1a6890 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:01:58.204768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3ab8e453-faae-4064-84eb-82bced1a6890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:58.337186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:01:58.368396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1094/2221] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:01:58.395148 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:58.396829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:58.541031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a707e6-2267-4e5a-af73-6a16fa24c8d6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:58.550165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1bbc91-d795-4bc7-a309-ae6d903227f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:58.551283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1bbc91-d795-4bc7-a309-ae6d903227f5 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:58.551544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1bbc91-d795-4bc7-a309-ae6d903227f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:58.551817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1bbc91-d795-4bc7-a309-ae6d903227f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:58.570797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1bbc91-d795-4bc7-a309-ae6d903227f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:58.577677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1bbc91-d795-4bc7-a309-ae6d903227f5 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:58.596923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1128/2222] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:01:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:01:59.538126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab8e453-faae-4064-84eb-82bced1a6890 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:01:59.540619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3ab8e453-faae-4064-84eb-82bced1a6890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:01:59.601371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab8e453-faae-4064-84eb-82bced1a6890 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:01:59.622675 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:01:59.624000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:59.624483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:59.624932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] PATH_INFO: `/v3/projects/33e1e7c36252488aa8a2255b3c44c498` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:01:59.626516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] RBAC: Authorizing `identity:delete_project(project_id=33e1e7c36252488aa8a2255b3c44c498)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:01:59.638039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:01:59.806630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33e1e7c36252488aa8a2255b3c44c498', 'request_id': 'req-745ba62b-5b0b-48b1-842a-851c3d30825c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:59.833286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33e1e7c36252488aa8a2255b3c44c498', 'request_id': 'req-745ba62b-5b0b-48b1-842a-851c3d30825c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:59.877434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33e1e7c36252488aa8a2255b3c44c498', 'request_id': 'req-745ba62b-5b0b-48b1-842a-851c3d30825c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:59.925168 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] The token cache is being invalidate because project 33e1e7c36252488aa8a2255b3c44c498 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:01:59.926790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:59.930900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-745ba62b-5b0b-48b1-842a-851c3d30825c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:01:59.946653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1095/2223] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:01:59 2022] DELETE /identity/v3/projects/33e1e7c36252488aa8a2255b3c44c498 => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:01:59.949637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82a627-3edd-485d-b9fe-1b8f1a75970f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:01:59.950620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82a627-3edd-485d-b9fe-1b8f1a75970f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:01:59.951081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82a627-3edd-485d-b9fe-1b8f1a75970f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:00.035047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-7d82a627-3edd-485d-b9fe-1b8f1a75970f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:00.041359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7d82a627-3edd-485d-b9fe-1b8f1a75970f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:00.173671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1129/2224] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:01:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:00.182596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53510571-c68d-46b4-a1fc-6a23a6c4c9ea None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:00.183508 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53510571-c68d-46b4-a1fc-6a23a6c4c9ea None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:00.183788 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53510571-c68d-46b4-a1fc-6a23a6c4c9ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:00.238597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-53510571-c68d-46b4-a1fc-6a23a6c4c9ea None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:00.239890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-53510571-c68d-46b4-a1fc-6a23a6c4c9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:00.278833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1096/2225] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:00.284433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82a627-3edd-485d-b9fe-1b8f1a75970f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:00.299205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82a627-3edd-485d-b9fe-1b8f1a75970f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:00.310375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:00.312972 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:00.314866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:00.315221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:00.316101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:00.318504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:00.320472 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:00.357024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1130/2226] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:00.361862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53510571-c68d-46b4-a1fc-6a23a6c4c9ea None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:00.372702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53510571-c68d-46b4-a1fc-6a23a6c4c9ea None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:00.397101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2265c12c-2aa3-425d-80ef-70cc64b57684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:00.398182 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265c12c-2aa3-425d-80ef-70cc64b57684 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:00.398600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265c12c-2aa3-425d-80ef-70cc64b57684 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:00.398985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265c12c-2aa3-425d-80ef-70cc64b57684 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:00.399749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2265c12c-2aa3-425d-80ef-70cc64b57684 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:00.402746 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2265c12c-2aa3-425d-80ef-70cc64b57684 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:00.474020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1097/2227] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:00 2022] POST /identity/v3/projects => generated 385 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:00.478800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:00.495167 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74abef1-2d9f-4dfb-a963-7173ea7d16f2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:00.576389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:00.720903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e022bb-3bd0-4bb7-a3e5-8700f6c1e034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:00.721582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e022bb-3bd0-4bb7-a3e5-8700f6c1e034 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:00.721863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e022bb-3bd0-4bb7-a3e5-8700f6c1e034 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:00.722139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e022bb-3bd0-4bb7-a3e5-8700f6c1e034 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:00.722784 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e022bb-3bd0-4bb7-a3e5-8700f6c1e034 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:00.724849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e022bb-3bd0-4bb7-a3e5-8700f6c1e034 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:00.814113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1131/2228] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:00 2022] POST /identity/v3/users => generated 293 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:00.815477 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2265c12c-2aa3-425d-80ef-70cc64b57684 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:00.832518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2265c12c-2aa3-425d-80ef-70cc64b57684 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:00.842972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70792af5-21a3-4540-a9e6-794e7ac9f9df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:00.843707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70792af5-21a3-4540-a9e6-794e7ac9f9df admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:00.844089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70792af5-21a3-4540-a9e6-794e7ac9f9df admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:00.847992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70792af5-21a3-4540-a9e6-794e7ac9f9df admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:00.851707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70792af5-21a3-4540-a9e6-794e7ac9f9df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:00.854419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70792af5-21a3-4540-a9e6-794e7ac9f9df admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:00.886629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e022bb-3bd0-4bb7-a3e5-8700f6c1e034 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:00.886873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1098/2229] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:00 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:00.920042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e022bb-3bd0-4bb7-a3e5-8700f6c1e034 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:00.921896 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:00.923601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:00.923601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:00.923601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:00.924023 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:00.925408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:00.975691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1132/2230] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:00 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:00.993705 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70792af5-21a3-4540-a9e6-794e7ac9f9df admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:01.027903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70792af5-21a3-4540-a9e6-794e7ac9f9df admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:01.037217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f9afb0-3602-401b-8544-1fbb2577121f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:01.039010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f9afb0-3602-401b-8544-1fbb2577121f admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:01.039696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f9afb0-3602-401b-8544-1fbb2577121f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:01.040358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f9afb0-3602-401b-8544-1fbb2577121f admin admin] PATH_INFO: `/v3/projects/7eedce7d7b9b4d4ebb254e75c2971918/users/f72f224dad444c9982d9ebc1783efd7e/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:01.113959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f9afb0-3602-401b-8544-1fbb2577121f admin admin] RBAC: Authorizing `identity:create_grant(project_id=7eedce7d7b9b4d4ebb254e75c2971918, user_id=f72f224dad444c9982d9ebc1783efd7e, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:01.133204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f9afb0-3602-401b-8544-1fbb2577121f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:01.159239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:01.161961 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:01.199462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1099/2231] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:00 2022] PUT /identity/v3/projects/7eedce7d7b9b4d4ebb254e75c2971918/users/f72f224dad444c9982d9ebc1783efd7e/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:01.199881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8b6309-d697-407d-96d9-9dc1c9abaafa admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:01.202794 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c0ade6-c557-4228-b005-53d24392d9b0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:01.203495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c0ade6-c557-4228-b005-53d24392d9b0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:01.203899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c0ade6-c557-4228-b005-53d24392d9b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:01.309753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c6c0ade6-c557-4228-b005-53d24392d9b0 None None] MFA Rules not processed for user `f72f224dad444c9982d9ebc1783efd7e`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:01.312219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c6c0ade6-c557-4228-b005-53d24392d9b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:01.470045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:01.472540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:01.472910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:01.473193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] PATH_INFO: `/v3/users/ebdef1d69bb14837b9a0c28cbba332fc` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:01.478891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ebdef1d69bb14837b9a0c28cbba332fc)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:01.489933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:01.512982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1100/2232] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:01.518445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c0ade6-c557-4228-b005-53d24392d9b0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:01.550926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:01.566697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c0ade6-c557-4228-b005-53d24392d9b0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:01.591733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fb8da6-6604-49ca-bbc7-09a3b82ab90a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:01.595021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fb8da6-6604-49ca-bbc7-09a3b82ab90a admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:01.595554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fb8da6-6604-49ca-bbc7-09a3b82ab90a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:01.596327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fb8da6-6604-49ca-bbc7-09a3b82ab90a admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:01.599010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fb8da6-6604-49ca-bbc7-09a3b82ab90a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:01.605458 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fb8da6-6604-49ca-bbc7-09a3b82ab90a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:01.659049 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ebdef1d69bb14837b9a0c28cbba332fc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:01.684818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ebdef1d69bb14837b9a0c28cbba332fc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:01.693406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1101/2233] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:01 2022] POST /identity/v3/projects => generated 383 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:01.695205 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fb8da6-6604-49ca-bbc7-09a3b82ab90a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:01.763832 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ebdef1d69bb14837b9a0c28cbba332fc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:01.765819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ebdef1d69bb14837b9a0c28cbba332fc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:01.769917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ebdef1d69bb14837b9a0c28cbba332fc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:01.824584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1133/2234] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:01 2022] DELETE /identity/v3/users/ebdef1d69bb14837b9a0c28cbba332fc => generated 0 bytes in 666 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:01.906093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fb8da6-6604-49ca-bbc7-09a3b82ab90a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:02.043016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5733b7-65f2-40e0-b7ef-0f699814bb57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:02.043748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5733b7-65f2-40e0-b7ef-0f699814bb57 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:02.044041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5733b7-65f2-40e0-b7ef-0f699814bb57 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:02.044346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5733b7-65f2-40e0-b7ef-0f699814bb57 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:02.045008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5733b7-65f2-40e0-b7ef-0f699814bb57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:02.047104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5733b7-65f2-40e0-b7ef-0f699814bb57 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:02.088270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1102/2235] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:01 2022] POST /identity/v3/users => generated 292 bytes in 391 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:02.089853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:02.091400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:02.114610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdab9c0-7881-4ee7-911c-ce83f0971d34 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:02.131838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3cb31f-fc15-466c-8d15-2c917fecb26c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:02.133662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3cb31f-fc15-466c-8d15-2c917fecb26c admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:02.145035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3cb31f-fc15-466c-8d15-2c917fecb26c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:02.146279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3cb31f-fc15-466c-8d15-2c917fecb26c admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:02.147388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3cb31f-fc15-466c-8d15-2c917fecb26c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:02.150940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3cb31f-fc15-466c-8d15-2c917fecb26c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:02.199217 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5733b7-65f2-40e0-b7ef-0f699814bb57 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:02.200567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1134/2236] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:02 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:02.210810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5733b7-65f2-40e0-b7ef-0f699814bb57 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:02.217558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff2add-88b4-4fad-83fd-ad565ae034cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:02.218767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff2add-88b4-4fad-83fd-ad565ae034cb admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:02.219263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff2add-88b4-4fad-83fd-ad565ae034cb admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:02.219722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff2add-88b4-4fad-83fd-ad565ae034cb admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:02.237692 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff2add-88b4-4fad-83fd-ad565ae034cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:02.243420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff2add-88b4-4fad-83fd-ad565ae034cb admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:02.275020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1103/2237] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:02 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:02.275963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3cb31f-fc15-466c-8d15-2c917fecb26c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:02.287236 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:02.295663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3cb31f-fc15-466c-8d15-2c917fecb26c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:02.304930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6109f2-3da2-4b4d-bb31-6983e4469724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:02.305918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6109f2-3da2-4b4d-bb31-6983e4469724 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:02.306504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6109f2-3da2-4b4d-bb31-6983e4469724 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:02.309637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6109f2-3da2-4b4d-bb31-6983e4469724 admin admin] PATH_INFO: `/v3/projects/d3403583d2964fc0969d69ce53bb7b6a/users/51a2347770114ead8b25b28bd858b7bd/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:02.349030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6109f2-3da2-4b4d-bb31-6983e4469724 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3403583d2964fc0969d69ce53bb7b6a, user_id=51a2347770114ead8b25b28bd858b7bd, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:02.353776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6109f2-3da2-4b4d-bb31-6983e4469724 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:02.431436 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1135/2238] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:02 2022] PUT /identity/v3/projects/d3403583d2964fc0969d69ce53bb7b6a/users/51a2347770114ead8b25b28bd858b7bd/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:02.433890 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad44d2-d02d-4c19-888b-035ff95d9dc0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:02.434179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad44d2-d02d-4c19-888b-035ff95d9dc0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:02.434466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad44d2-d02d-4c19-888b-035ff95d9dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:02.494129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e0ad44d2-d02d-4c19-888b-035ff95d9dc0 None None] MFA Rules not processed for user `51a2347770114ead8b25b28bd858b7bd`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:02.496014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e0ad44d2-d02d-4c19-888b-035ff95d9dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:02.586673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:02.601858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1104/2239] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:02 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:02.606473 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6109f2-3da2-4b4d-bb31-6983e4469724 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:02.726509 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6109f2-3da2-4b4d-bb31-6983e4469724 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:02.739461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0650e7-c630-40c5-b1fd-392d80ace892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:02.740456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0650e7-c630-40c5-b1fd-392d80ace892 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:02.742791 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0650e7-c630-40c5-b1fd-392d80ace892 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:02.743204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0650e7-c630-40c5-b1fd-392d80ace892 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:02.746772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0650e7-c630-40c5-b1fd-392d80ace892 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:02.751482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0650e7-c630-40c5-b1fd-392d80ace892 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:02.850879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1136/2240] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:02 2022] POST /identity/v3/projects => generated 385 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:02.851743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad44d2-d02d-4c19-888b-035ff95d9dc0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:02.872289 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad44d2-d02d-4c19-888b-035ff95d9dc0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:03.035165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a37f345-86c0-4fbd-9cf8-647bfdf37d47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:03.035809 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37f345-86c0-4fbd-9cf8-647bfdf37d47 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:03.036094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37f345-86c0-4fbd-9cf8-647bfdf37d47 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:03.036401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37f345-86c0-4fbd-9cf8-647bfdf37d47 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:03.037132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a37f345-86c0-4fbd-9cf8-647bfdf37d47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:03.046575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a37f345-86c0-4fbd-9cf8-647bfdf37d47 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:03.096499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1105/2241] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:02 2022] POST /identity/v3/users => generated 299 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:03.097786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0650e7-c630-40c5-b1fd-392d80ace892 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:03.118771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0650e7-c630-40c5-b1fd-392d80ace892 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:03.130873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9c12ee-38f9-4c16-b8e5-7aa513419810 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:03.131708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9c12ee-38f9-4c16-b8e5-7aa513419810 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:03.135550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9c12ee-38f9-4c16-b8e5-7aa513419810 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:03.135992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9c12ee-38f9-4c16-b8e5-7aa513419810 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:03.137080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9c12ee-38f9-4c16-b8e5-7aa513419810 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:03.141093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9c12ee-38f9-4c16-b8e5-7aa513419810 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:03.193079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1137/2242] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:03 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:03.195136 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a37f345-86c0-4fbd-9cf8-647bfdf37d47 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:03.208515 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a37f345-86c0-4fbd-9cf8-647bfdf37d47 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:03.218499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:03.220444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:03.220914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:03.221363 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] PATH_INFO: `/v3/projects/5448fb5300304d38b9734af07c3ba834/users/b48ce49931ba40fd9d553aa9a12b4ddd/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:03.249903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] RBAC: Authorizing `identity:create_grant(project_id=5448fb5300304d38b9734af07c3ba834, user_id=b48ce49931ba40fd9d553aa9a12b4ddd, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:03.253158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:03.284597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1106/2243] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:03 2022] PUT /identity/v3/projects/5448fb5300304d38b9734af07c3ba834/users/b48ce49931ba40fd9d553aa9a12b4ddd/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:03.285939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9c12ee-38f9-4c16-b8e5-7aa513419810 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:03.302523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9c12ee-38f9-4c16-b8e5-7aa513419810 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:03.565012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:03.566115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:03.567877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3b3696-2cb2-4af2-a2e8-d0c128e2fb74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:03.569422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3b3696-2cb2-4af2-a2e8-d0c128e2fb74 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:03.570394 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3b3696-2cb2-4af2-a2e8-d0c128e2fb74 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:03.571780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3b3696-2cb2-4af2-a2e8-d0c128e2fb74 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:03.575063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3b3696-2cb2-4af2-a2e8-d0c128e2fb74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:03.578630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3b3696-2cb2-4af2-a2e8-d0c128e2fb74 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:03.593831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7d9939-48c3-4568-ae0c-1831071dd75b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:03.609177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1138/2244] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:03 2022] GET /identity/v3/roles => generated 1420 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:03.613025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:03.614042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:03.614369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:03.614634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] PATH_INFO: `/v3/projects/b34f10ec378a4622a8f0b0546e84da7f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:03.615692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3b3696-2cb2-4af2-a2e8-d0c128e2fb74 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:03.619099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b34f10ec378a4622a8f0b0546e84da7f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:03.626751 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:03.661012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3b3696-2cb2-4af2-a2e8-d0c128e2fb74 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:03.676720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c921d4-5554-40b2-8619-e33015bd72b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:03.681635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c921d4-5554-40b2-8619-e33015bd72b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:03.686375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c921d4-5554-40b2-8619-e33015bd72b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:03.687373 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c921d4-5554-40b2-8619-e33015bd72b6 admin admin] PATH_INFO: `/v3/projects/5448fb5300304d38b9734af07c3ba834/users/b48ce49931ba40fd9d553aa9a12b4ddd/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:03.702179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c921d4-5554-40b2-8619-e33015bd72b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5448fb5300304d38b9734af07c3ba834, user_id=b48ce49931ba40fd9d553aa9a12b4ddd, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:03.753353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c921d4-5554-40b2-8619-e33015bd72b6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:03.782631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b34f10ec378a4622a8f0b0546e84da7f', 'request_id': 'req-3acd866b-a03a-46ec-8993-1446bd026464'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:03.785410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1139/2245] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:03 2022] PUT /identity/v3/projects/5448fb5300304d38b9734af07c3ba834/users/b48ce49931ba40fd9d553aa9a12b4ddd/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:03.800378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5326-2aec-47fb-bd17-aa617e3440cb None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:03.800879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b34f10ec378a4622a8f0b0546e84da7f', 'request_id': 'req-3acd866b-a03a-46ec-8993-1446bd026464'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:03.802591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5326-2aec-47fb-bd17-aa617e3440cb None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:03.811108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b34f10ec378a4622a8f0b0546e84da7f', 'request_id': 'req-3acd866b-a03a-46ec-8993-1446bd026464'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:03.812663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5326-2aec-47fb-bd17-aa617e3440cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:03.831857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] The token cache is being invalidate because project b34f10ec378a4622a8f0b0546e84da7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:03.832391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:03.833397 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3acd866b-a03a-46ec-8993-1446bd026464 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:03.841090 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1107/2246] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:03 2022] DELETE /identity/v3/projects/b34f10ec378a4622a8f0b0546e84da7f => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:03.842902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92575e02-8071-4286-8bba-93dd5a1af333 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:03.843254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92575e02-8071-4286-8bba-93dd5a1af333 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:03.843669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92575e02-8071-4286-8bba-93dd5a1af333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:03.868473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-92575e02-8071-4286-8bba-93dd5a1af333 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:03.870130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-92575e02-8071-4286-8bba-93dd5a1af333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:03.913677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-58cf5326-2aec-47fb-bd17-aa617e3440cb None None] MFA Rules not processed for user `b48ce49931ba40fd9d553aa9a12b4ddd`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:03.914967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-58cf5326-2aec-47fb-bd17-aa617e3440cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:03.946865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:03.995210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:04.015197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1108/2247] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:04.017511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca5de8a-3ddd-4d6f-a37a-46f447cb62c6 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.018213 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca5de8a-3ddd-4d6f-a37a-46f447cb62c6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.018599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca5de8a-3ddd-4d6f-a37a-46f447cb62c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.051809 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-aca5de8a-3ddd-4d6f-a37a-46f447cb62c6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:04.053646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-aca5de8a-3ddd-4d6f-a37a-46f447cb62c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:04.070433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1109/2248] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:04.076653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca5de8a-3ddd-4d6f-a37a-46f447cb62c6 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:04.092024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca5de8a-3ddd-4d6f-a37a-46f447cb62c6 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:04.098258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:04.099202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.099595 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.100414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.101265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:04.101682 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:04.103623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:04.113459 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1140/2249] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:03 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:04.126957 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cf5326-2aec-47fb-bd17-aa617e3440cb None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:04.128852 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-58cf5326-2aec-47fb-bd17-aa617e3440cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:04.131935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1110/2250] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:04.133006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:04.146667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e5b763-681a-4fb0-91ee-23ae7c86285e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:04.153198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a85d0-e72d-48b4-a783-2ad1d7022f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:04.154193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a85d0-e72d-48b4-a783-2ad1d7022f62 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.154626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a85d0-e72d-48b4-a783-2ad1d7022f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.155071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a85d0-e72d-48b4-a783-2ad1d7022f62 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.155826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756a85d0-e72d-48b4-a783-2ad1d7022f62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:04.159081 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756a85d0-e72d-48b4-a783-2ad1d7022f62 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:04.201248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1111/2251] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:04 2022] POST /identity/v3/projects => generated 373 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:04.206143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a85d0-e72d-48b4-a783-2ad1d7022f62 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:04.339911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cf5326-2aec-47fb-bd17-aa617e3440cb None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:04.366317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a85d0-e72d-48b4-a783-2ad1d7022f62 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:04.383956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8250139c-67f6-4afd-833e-63f24136c07d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:04.384741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8250139c-67f6-4afd-833e-63f24136c07d admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.385002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8250139c-67f6-4afd-833e-63f24136c07d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.385279 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8250139c-67f6-4afd-833e-63f24136c07d admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.386836 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8250139c-67f6-4afd-833e-63f24136c07d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:04.390321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8250139c-67f6-4afd-833e-63f24136c07d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:04.440468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1112/2252] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:04 2022] POST /identity/v3/users => generated 287 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:04.441414 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8250139c-67f6-4afd-833e-63f24136c07d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:04.449648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb5280b-7487-4cd4-8c33-30296e546126 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:04.450788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5280b-7487-4cd4-8c33-30296e546126 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.450919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5280b-7487-4cd4-8c33-30296e546126 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.451186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5280b-7487-4cd4-8c33-30296e546126 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.453465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-bbb5280b-7487-4cd4-8c33-30296e546126 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:04.458962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8250139c-67f6-4afd-833e-63f24136c07d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:04.479523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084c75a7-8702-4c96-b662-904cacaada2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:04.480250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084c75a7-8702-4c96-b662-904cacaada2b admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.480594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084c75a7-8702-4c96-b662-904cacaada2b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.480870 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084c75a7-8702-4c96-b662-904cacaada2b admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.481539 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084c75a7-8702-4c96-b662-904cacaada2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:04.483193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084c75a7-8702-4c96-b662-904cacaada2b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:04.503601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1113/2253] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:04.510346 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084c75a7-8702-4c96-b662-904cacaada2b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:04.537149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084c75a7-8702-4c96-b662-904cacaada2b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:04.548922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e70af62-e876-4ffb-8d83-14456957ea4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:04.549571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e70af62-e876-4ffb-8d83-14456957ea4a admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.549924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e70af62-e876-4ffb-8d83-14456957ea4a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.550141 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e70af62-e876-4ffb-8d83-14456957ea4a admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.550837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e70af62-e876-4ffb-8d83-14456957ea4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:04.552711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e70af62-e876-4ffb-8d83-14456957ea4a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:04.575374 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1114/2254] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:04.583610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e70af62-e876-4ffb-8d83-14456957ea4a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:04.609898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e70af62-e876-4ffb-8d83-14456957ea4a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:04.614125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb5280b-7487-4cd4-8c33-30296e546126 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:04.620164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb5280b-7487-4cd4-8c33-30296e546126 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:04.626001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:04.626676 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.626957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.627238 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] PATH_INFO: `/v3/projects/6762a2261e54498e82855f614d98562f/users/661867b9dc1541d18d5224cc292ab010/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.637703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1141/2255] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 511 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:04.663976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6762a2261e54498e82855f614d98562f, user_id=661867b9dc1541d18d5224cc292ab010, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:04.666762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:04.696380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1115/2256] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:04 2022] PUT /identity/v3/projects/6762a2261e54498e82855f614d98562f/users/661867b9dc1541d18d5224cc292ab010/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:04.711101 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc14e8-0865-4aee-b61a-ace625e10f99 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:04.711998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc14e8-0865-4aee-b61a-ace625e10f99 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:04.712670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc14e8-0865-4aee-b61a-ace625e10f99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:04.772734 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-5fcc14e8-0865-4aee-b61a-ace625e10f99 None None] MFA Rules not processed for user `661867b9dc1541d18d5224cc292ab010`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:04.777853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5fcc14e8-0865-4aee-b61a-ace625e10f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:04.829656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:04.931682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1142/2257] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:04 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:04.951811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:04.954445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:05.039509 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:05.074831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3129922-6fad-4a2b-a8ff-ad9b12a4c763 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:05.102189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:05.103429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:05.103429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:05.103720 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:05.133602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:05.135029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:05.157393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1116/2258] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:04 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:06.407844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc14e8-0865-4aee-b61a-ace625e10f99 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:06.409467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-5fcc14e8-0865-4aee-b61a-ace625e10f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:06.607826 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc14e8-0865-4aee-b61a-ace625e10f99 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:06.613210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:06.613877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:06.614140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:06.614465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] PATH_INFO: `/v3/users/552c23b8456a40ec8e01297aca7965d3` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:06.615860 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=552c23b8456a40ec8e01297aca7965d3)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:06.617978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:06.686353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '552c23b8456a40ec8e01297aca7965d3'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:06.694273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '552c23b8456a40ec8e01297aca7965d3'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:06.707618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '552c23b8456a40ec8e01297aca7965d3'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:06.708637 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '552c23b8456a40ec8e01297aca7965d3'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:06.709333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '552c23b8456a40ec8e01297aca7965d3'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:06.723816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1143/2259] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:06 2022] DELETE /identity/v3/users/552c23b8456a40ec8e01297aca7965d3 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:06.793949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:06.795545 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:06.857306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64359402-6d2c-4219-9758-4019977bf155 service nova] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:06.923245 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ea7c78-22b5-4d61-a2b5-034c1105c071 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:06.923919 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea7c78-22b5-4d61-a2b5-034c1105c071 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:06.924297 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea7c78-22b5-4d61-a2b5-034c1105c071 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:06.924636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea7c78-22b5-4d61-a2b5-034c1105c071 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:06.926810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b7ea7c78-22b5-4d61-a2b5-034c1105c071 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:06.932928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:06.934622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:07.018379 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:07.029561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ea7c78-22b5-4d61-a2b5-034c1105c071 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.031010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ea7c78-22b5-4d61-a2b5-034c1105c071 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.040764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05087766-d30c-4943-be3d-4efeae4d8f7b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.041599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1117/2260] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:07.045727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.046410 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.046640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.046910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] PATH_INFO: `/v3/projects/7ff48b26c576495f8bb8c88d20d246cc` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.048799 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ff48b26c576495f8bb8c88d20d246cc)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.050940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.104984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ff48b26c576495f8bb8c88d20d246cc', 'request_id': 'req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:07.114171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ff48b26c576495f8bb8c88d20d246cc', 'request_id': 'req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:07.126208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ff48b26c576495f8bb8c88d20d246cc', 'request_id': 'req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:07.143378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] The token cache is being invalidate because project 7ff48b26c576495f8bb8c88d20d246cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:07.143691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:07.144421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-2691193c-e9b7-4d95-94cb-ef36df9b4f4b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:07.151606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1144/2261] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:06 2022] DELETE /identity/v3/projects/7ff48b26c576495f8bb8c88d20d246cc => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:07.153099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60075921-1e13-4fa3-8fb5-a2a08e801e10 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.153396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60075921-1e13-4fa3-8fb5-a2a08e801e10 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.153678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60075921-1e13-4fa3-8fb5-a2a08e801e10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.178233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-60075921-1e13-4fa3-8fb5-a2a08e801e10 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:07.179535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-60075921-1e13-4fa3-8fb5-a2a08e801e10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:07.236157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:07.273749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1118/2262] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:07.275490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350a626-3e0a-4b3a-b2bc-22692e415d12 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.275811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350a626-3e0a-4b3a-b2bc-22692e415d12 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.276108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350a626-3e0a-4b3a-b2bc-22692e415d12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.299910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d350a626-3e0a-4b3a-b2bc-22692e415d12 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:07.301914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d350a626-3e0a-4b3a-b2bc-22692e415d12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:07.313034 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1145/2263] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:07.317358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60075921-1e13-4fa3-8fb5-a2a08e801e10 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:07.329470 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60075921-1e13-4fa3-8fb5-a2a08e801e10 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.334772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.335534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.335880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.336275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.336981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:07.337352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.339151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.352183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1119/2264] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:07.357380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d350a626-3e0a-4b3a-b2bc-22692e415d12 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:07.370108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d350a626-3e0a-4b3a-b2bc-22692e415d12 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.376956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50edc596-142b-4966-b98c-4fdffc1ba526 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.378632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50edc596-142b-4966-b98c-4fdffc1ba526 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.378954 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50edc596-142b-4966-b98c-4fdffc1ba526 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.379230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50edc596-142b-4966-b98c-4fdffc1ba526 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.379920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50edc596-142b-4966-b98c-4fdffc1ba526 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.382008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50edc596-142b-4966-b98c-4fdffc1ba526 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.412456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1146/2265] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:07 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:07.416483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:07.427390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21781581-7232-4593-a15b-89e24f448717 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.488782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251c722-ea77-4cac-8571-dc818061f7aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.489388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251c722-ea77-4cac-8571-dc818061f7aa admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.489667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251c722-ea77-4cac-8571-dc818061f7aa admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.489935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251c722-ea77-4cac-8571-dc818061f7aa admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.490957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251c722-ea77-4cac-8571-dc818061f7aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.493559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251c722-ea77-4cac-8571-dc818061f7aa admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.519737 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1120/2266] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:07 2022] POST /identity/v3/users => generated 293 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:07.524491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50edc596-142b-4966-b98c-4fdffc1ba526 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:07.537984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50edc596-142b-4966-b98c-4fdffc1ba526 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.543945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bd2e93-7d34-4ab9-834d-c34f4c0ed242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.544854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bd2e93-7d34-4ab9-834d-c34f4c0ed242 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.545340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bd2e93-7d34-4ab9-834d-c34f4c0ed242 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.545533 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bd2e93-7d34-4ab9-834d-c34f4c0ed242 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.546293 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bd2e93-7d34-4ab9-834d-c34f4c0ed242 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.548012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bd2e93-7d34-4ab9-834d-c34f4c0ed242 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.564170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1147/2267] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:07.565670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251c722-ea77-4cac-8571-dc818061f7aa admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:07.577962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251c722-ea77-4cac-8571-dc818061f7aa admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.585202 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d75135-1c3a-4446-9163-f90b808af23c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.585835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d75135-1c3a-4446-9163-f90b808af23c admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.586222 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d75135-1c3a-4446-9163-f90b808af23c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.586600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d75135-1c3a-4446-9163-f90b808af23c admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.587306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d75135-1c3a-4446-9163-f90b808af23c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.589026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d75135-1c3a-4446-9163-f90b808af23c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.619405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1121/2268] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:07.623796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bd2e93-7d34-4ab9-834d-c34f4c0ed242 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:07.634127 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bd2e93-7d34-4ab9-834d-c34f4c0ed242 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.640228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.640351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.640581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.640845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] PATH_INFO: `/v3/projects/51f1284fd166460bbd7243217b6da7d1/users/5e063f7f37ae43cdb2016f3bab247184/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.672404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=51f1284fd166460bbd7243217b6da7d1, user_id=5e063f7f37ae43cdb2016f3bab247184, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.672404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.679225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1148/2269] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:07 2022] PUT /identity/v3/projects/51f1284fd166460bbd7243217b6da7d1/users/5e063f7f37ae43cdb2016f3bab247184/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:07.690552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006054c-e5e7-4b09-b75e-9c4b9a354edf None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.690831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006054c-e5e7-4b09-b75e-9c4b9a354edf None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.691102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006054c-e5e7-4b09-b75e-9c4b9a354edf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.723353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c006054c-e5e7-4b09-b75e-9c4b9a354edf None None] MFA Rules not processed for user `5e063f7f37ae43cdb2016f3bab247184`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:07.724882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c006054c-e5e7-4b09-b75e-9c4b9a354edf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:07.764365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:07.766143 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:07.812896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1122/2270] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:07.823918 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006054c-e5e7-4b09-b75e-9c4b9a354edf None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:07.826709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c006054c-e5e7-4b09-b75e-9c4b9a354edf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:07.829997 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d9a643-4bb5-4161-ba7c-edda9691b9ce admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.833753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.834350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.834629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.834896 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.836660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:07.859968 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:07.870685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:07.904704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006054c-e5e7-4b09-b75e-9c4b9a354edf None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:07.907147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.909682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.910724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6df198-7b2f-4233-b5d3-2639309118f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:07.911334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6df198-7b2f-4233-b5d3-2639309118f7 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:07.911619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6df198-7b2f-4233-b5d3-2639309118f7 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:07.911892 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6df198-7b2f-4233-b5d3-2639309118f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:07.920605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6df198-7b2f-4233-b5d3-2639309118f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:07.922264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6df198-7b2f-4233-b5d3-2639309118f7 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:07.923129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1149/2271] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:07 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:07.934757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1123/2272] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:08.456595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:08.456595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:08.519853 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0c3015-70f4-4f92-a5cb-2a07a4389fc5 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:08.524408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:08.525022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:08.525297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:08.525565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] PATH_INFO: `/v3/users/661867b9dc1541d18d5224cc292ab010` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:08.526706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=661867b9dc1541d18d5224cc292ab010)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:08.528746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:08.578290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '661867b9dc1541d18d5224cc292ab010'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:08.585281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '661867b9dc1541d18d5224cc292ab010'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:08.597699 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '661867b9dc1541d18d5224cc292ab010'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:08.598404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '661867b9dc1541d18d5224cc292ab010'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:08.599012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '661867b9dc1541d18d5224cc292ab010'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:08.611756 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1150/2273] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:08 2022] DELETE /identity/v3/users/661867b9dc1541d18d5224cc292ab010 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:08.616537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83f8192-1bf3-4cfd-84e2-d49d966a8dd8 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:08.616880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83f8192-1bf3-4cfd-84e2-d49d966a8dd8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:08.617092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83f8192-1bf3-4cfd-84e2-d49d966a8dd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:08.638082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c83f8192-1bf3-4cfd-84e2-d49d966a8dd8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:08.639287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c83f8192-1bf3-4cfd-84e2-d49d966a8dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:08.698987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1124/2274] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:08.709538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:08.710975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:08.753564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:08.770286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26bd791-0984-482c-ad96-406a77206a64 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:08.773928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6b6c25-280d-4140-a419-b1eb1be4de09 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:08.774523 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b6c25-280d-4140-a419-b1eb1be4de09 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:08.774798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b6c25-280d-4140-a419-b1eb1be4de09 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:08.775066 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b6c25-280d-4140-a419-b1eb1be4de09 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:08.785098 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6b6c25-280d-4140-a419-b1eb1be4de09 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:08.786525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6b6c25-280d-4140-a419-b1eb1be4de09 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:08.796739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1151/2275] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:09.214067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83f8192-1bf3-4cfd-84e2-d49d966a8dd8 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:09.217391 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c83f8192-1bf3-4cfd-84e2-d49d966a8dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:09.239579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83f8192-1bf3-4cfd-84e2-d49d966a8dd8 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:09.249358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:09.249944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.250415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.250664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] PATH_INFO: `/v3/projects/6762a2261e54498e82855f614d98562f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.252029 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] RBAC: Authorizing `identity:delete_project(project_id=6762a2261e54498e82855f614d98562f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:09.254167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:09.308426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6762a2261e54498e82855f614d98562f', 'request_id': 'req-61bd2a43-8366-4af4-a4d5-165ae592e89e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:09.314882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6762a2261e54498e82855f614d98562f', 'request_id': 'req-61bd2a43-8366-4af4-a4d5-165ae592e89e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:09.323369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6762a2261e54498e82855f614d98562f', 'request_id': 'req-61bd2a43-8366-4af4-a4d5-165ae592e89e'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:09.339288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] The token cache is being invalidate because project 6762a2261e54498e82855f614d98562f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:09.340370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:09.349593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-61bd2a43-8366-4af4-a4d5-165ae592e89e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:09.351290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1125/2276] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:09 2022] DELETE /identity/v3/projects/6762a2261e54498e82855f614d98562f => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:09.356582 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea04df-75fe-4f2c-90c1-8f8a22bf2aca None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.357152 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea04df-75fe-4f2c-90c1-8f8a22bf2aca None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.358525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea04df-75fe-4f2c-90c1-8f8a22bf2aca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.389774 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e2ea04df-75fe-4f2c-90c1-8f8a22bf2aca None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:09.390979 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e2ea04df-75fe-4f2c-90c1-8f8a22bf2aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:09.468244 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1152/2277] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:09.473098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e573eb-851f-4c3c-ab3d-1be501ef2f3c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.473384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e573eb-851f-4c3c-ab3d-1be501ef2f3c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.473660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e573eb-851f-4c3c-ab3d-1be501ef2f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.495875 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-30e573eb-851f-4c3c-ab3d-1be501ef2f3c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:09.498241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-30e573eb-851f-4c3c-ab3d-1be501ef2f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:09.507307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1126/2278] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:09.512978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ea04df-75fe-4f2c-90c1-8f8a22bf2aca None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:09.522599 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ea04df-75fe-4f2c-90c1-8f8a22bf2aca None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:09.528870 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:09.529573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.529873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.530145 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.530797 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:09.531093 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:09.532717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:09.546465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1153/2279] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:09.551575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e573eb-851f-4c3c-ab3d-1be501ef2f3c None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:09.562530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e573eb-851f-4c3c-ab3d-1be501ef2f3c None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:09.567188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0230724-432e-4b24-923b-7f412d955758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:09.567800 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230724-432e-4b24-923b-7f412d955758 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.568079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230724-432e-4b24-923b-7f412d955758 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.568382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230724-432e-4b24-923b-7f412d955758 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.569067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0230724-432e-4b24-923b-7f412d955758 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:09.571305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0230724-432e-4b24-923b-7f412d955758 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:09.601903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1127/2280] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:09 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:09.606552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:09.619302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65afe1fd-ec2e-48fb-8dad-ab2ce7f5a033 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:09.690706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd808e1-7396-4ddc-a6c8-02728b831a17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:09.691322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd808e1-7396-4ddc-a6c8-02728b831a17 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.691604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd808e1-7396-4ddc-a6c8-02728b831a17 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.691875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd808e1-7396-4ddc-a6c8-02728b831a17 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.692541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd808e1-7396-4ddc-a6c8-02728b831a17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:09.694475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd808e1-7396-4ddc-a6c8-02728b831a17 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:09.723351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1154/2281] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:09 2022] POST /identity/v3/users => generated 291 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:09.724860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0230724-432e-4b24-923b-7f412d955758 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:09.736087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0230724-432e-4b24-923b-7f412d955758 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:09.741439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920e9a82-56f2-4d98-b076-7934f18d2576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:09.741867 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920e9a82-56f2-4d98-b076-7934f18d2576 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.742151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920e9a82-56f2-4d98-b076-7934f18d2576 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.742418 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920e9a82-56f2-4d98-b076-7934f18d2576 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.743066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920e9a82-56f2-4d98-b076-7934f18d2576 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:09.744721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920e9a82-56f2-4d98-b076-7934f18d2576 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:09.759238 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1128/2282] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:09.760444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd808e1-7396-4ddc-a6c8-02728b831a17 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:09.772607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd808e1-7396-4ddc-a6c8-02728b831a17 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:09.778244 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d894a2a-ebd3-4f11-9885-38be9b3034bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:09.779012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d894a2a-ebd3-4f11-9885-38be9b3034bd admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.779291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d894a2a-ebd3-4f11-9885-38be9b3034bd admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.779560 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d894a2a-ebd3-4f11-9885-38be9b3034bd admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.780230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d894a2a-ebd3-4f11-9885-38be9b3034bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:09.781809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d894a2a-ebd3-4f11-9885-38be9b3034bd admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:09.792895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1155/2283] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:09 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:09.800774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920e9a82-56f2-4d98-b076-7934f18d2576 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:09.812404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920e9a82-56f2-4d98-b076-7934f18d2576 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:09.817197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed3ee37-53e4-42f4-b84c-bb5e02220eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:09.817848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3ee37-53e4-42f4-b84c-bb5e02220eb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.818130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3ee37-53e4-42f4-b84c-bb5e02220eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.818402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3ee37-53e4-42f4-b84c-bb5e02220eb6 admin admin] PATH_INFO: `/v3/projects/082e59576f304acb9858b1fa2605344d/users/caed935a15c34fd888ce2974aa28839b/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.820857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:09.837896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed3ee37-53e4-42f4-b84c-bb5e02220eb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=082e59576f304acb9858b1fa2605344d, user_id=caed935a15c34fd888ce2974aa28839b, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:09.841434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed3ee37-53e4-42f4-b84c-bb5e02220eb6 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:09.855077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1129/2284] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:09 2022] PUT /identity/v3/projects/082e59576f304acb9858b1fa2605344d/users/caed935a15c34fd888ce2974aa28839b/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:09.865907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1daf3e2-f7ea-4588-8c6a-b32fa0a84543 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:09.866197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1daf3e2-f7ea-4588-8c6a-b32fa0a84543 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:09.866463 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1daf3e2-f7ea-4588-8c6a-b32fa0a84543 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:09.895512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e1daf3e2-f7ea-4588-8c6a-b32fa0a84543 None None] MFA Rules not processed for user `caed935a15c34fd888ce2974aa28839b`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:09.896759 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e1daf3e2-f7ea-4588-8c6a-b32fa0a84543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:09.929726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:09.963309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1156/2285] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:09 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:09.970209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed3ee37-53e4-42f4-b84c-bb5e02220eb6 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:10.044167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed3ee37-53e4-42f4-b84c-bb5e02220eb6 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:10.049198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867d8e08-5f56-44dc-826a-21c7274860bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:10.049806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867d8e08-5f56-44dc-826a-21c7274860bd admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.050084 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867d8e08-5f56-44dc-826a-21c7274860bd admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.050355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867d8e08-5f56-44dc-826a-21c7274860bd admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.050999 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867d8e08-5f56-44dc-826a-21c7274860bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:10.052941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867d8e08-5f56-44dc-826a-21c7274860bd admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:10.085891 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1130/2286] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:09 2022] POST /identity/v3/projects => generated 381 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:10.087069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1daf3e2-f7ea-4588-8c6a-b32fa0a84543 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:10.098551 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1daf3e2-f7ea-4588-8c6a-b32fa0a84543 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:10.166120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226bd76a-4780-4d2b-b194-77cd4239d713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:10.166779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226bd76a-4780-4d2b-b194-77cd4239d713 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.167070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226bd76a-4780-4d2b-b194-77cd4239d713 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.167594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226bd76a-4780-4d2b-b194-77cd4239d713 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.168412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226bd76a-4780-4d2b-b194-77cd4239d713 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:10.170329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226bd76a-4780-4d2b-b194-77cd4239d713 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:10.194425 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1157/2287] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:10 2022] POST /identity/v3/users => generated 297 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:10.198417 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867d8e08-5f56-44dc-826a-21c7274860bd admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:10.209936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867d8e08-5f56-44dc-826a-21c7274860bd admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:10.215099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be69bb-b234-4533-b289-fa17f6993363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:10.215740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be69bb-b234-4533-b289-fa17f6993363 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.215864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be69bb-b234-4533-b289-fa17f6993363 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.216160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be69bb-b234-4533-b289-fa17f6993363 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.216810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be69bb-b234-4533-b289-fa17f6993363 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:10.218421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be69bb-b234-4533-b289-fa17f6993363 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:10.230129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1131/2288] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:10.234464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226bd76a-4780-4d2b-b194-77cd4239d713 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:10.246426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226bd76a-4780-4d2b-b194-77cd4239d713 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:10.251578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52069f3-4d4b-4c01-8c94-9138f70aa173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:10.252297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52069f3-4d4b-4c01-8c94-9138f70aa173 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.252584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52069f3-4d4b-4c01-8c94-9138f70aa173 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.252860 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52069f3-4d4b-4c01-8c94-9138f70aa173 admin admin] PATH_INFO: `/v3/projects/6bd809b216bd4f6994af711fbd30dcec/users/2e79cf35ad454c63aa593ad1782ba73a/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.269009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52069f3-4d4b-4c01-8c94-9138f70aa173 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bd809b216bd4f6994af711fbd30dcec, user_id=2e79cf35ad454c63aa593ad1782ba73a, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:10.271660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52069f3-4d4b-4c01-8c94-9138f70aa173 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:10.284336 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1158/2289] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:10 2022] PUT /identity/v3/projects/6bd809b216bd4f6994af711fbd30dcec/users/2e79cf35ad454c63aa593ad1782ba73a/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:10.288457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be69bb-b234-4533-b289-fa17f6993363 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:10.298422 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be69bb-b234-4533-b289-fa17f6993363 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:10.354726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d62016-9fdc-4ab8-a5af-9a73ac0cf299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:10.356073 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d62016-9fdc-4ab8-a5af-9a73ac0cf299 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.356574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d62016-9fdc-4ab8-a5af-9a73ac0cf299 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.356873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d62016-9fdc-4ab8-a5af-9a73ac0cf299 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.357505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d62016-9fdc-4ab8-a5af-9a73ac0cf299 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:10.359121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d62016-9fdc-4ab8-a5af-9a73ac0cf299 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:10.369456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1132/2290] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:10 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:10.374484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52069f3-4d4b-4c01-8c94-9138f70aa173 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:10.387050 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52069f3-4d4b-4c01-8c94-9138f70aa173 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:10.392024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:10.392711 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.392990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.393262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] PATH_INFO: `/v3/projects/6bd809b216bd4f6994af711fbd30dcec/users/2e79cf35ad454c63aa593ad1782ba73a/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.395233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bd809b216bd4f6994af711fbd30dcec, user_id=2e79cf35ad454c63aa593ad1782ba73a, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:10.397899 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:10.408626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1159/2291] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:10 2022] PUT /identity/v3/projects/6bd809b216bd4f6994af711fbd30dcec/users/2e79cf35ad454c63aa593ad1782ba73a/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:10.421080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7ea1e8-8c4f-41e9-be32-ddce9733c1a0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.421367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7ea1e8-8c4f-41e9-be32-ddce9733c1a0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.421638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7ea1e8-8c4f-41e9-be32-ddce9733c1a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.450272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2f7ea1e8-8c4f-41e9-be32-ddce9733c1a0 None None] MFA Rules not processed for user `2e79cf35ad454c63aa593ad1782ba73a`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:10.451533 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2f7ea1e8-8c4f-41e9-be32-ddce9733c1a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:10.500577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:10.537524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1133/2292] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:10 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:10.549668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:10.551449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:10.613212 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99a17d8-7b5f-433f-b6fa-85ce6e8da7c7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:10.617158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:10.617786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.618055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.618328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.628557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:10.629991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:10.642669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1160/2293] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:10 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:10.723935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7ea1e8-8c4f-41e9-be32-ddce9733c1a0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:10.735992 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7ea1e8-8c4f-41e9-be32-ddce9733c1a0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:10.739847 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cfb7f6-0277-495e-986c-71f36ac2569c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:10.740901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfb7f6-0277-495e-986c-71f36ac2569c service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:10.741204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfb7f6-0277-495e-986c-71f36ac2569c service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:10.741478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfb7f6-0277-495e-986c-71f36ac2569c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:10.811899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cfb7f6-0277-495e-986c-71f36ac2569c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:10.813405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cfb7f6-0277-495e-986c-71f36ac2569c service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:10.823543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1134/2294] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:10 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:15.099040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:15.100492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:15.141421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:15.174640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f98c51c-4918-4f26-900d-ca3894b3699a service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:15.179318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:15.179941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:15.180253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:15.180528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] PATH_INFO: `/v3/users/f72f224dad444c9982d9ebc1783efd7e` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:15.181725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f72f224dad444c9982d9ebc1783efd7e)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:15.184154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:15.232325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f72f224dad444c9982d9ebc1783efd7e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.239358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f72f224dad444c9982d9ebc1783efd7e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.252210 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f72f224dad444c9982d9ebc1783efd7e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.252930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f72f224dad444c9982d9ebc1783efd7e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.253543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f72f224dad444c9982d9ebc1783efd7e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.268915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1161/2295] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:15 2022] DELETE /identity/v3/users/f72f224dad444c9982d9ebc1783efd7e => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:15.273505 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b41e52a-bd2f-44d2-996c-b1d052ba47f8 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:15.274009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b41e52a-bd2f-44d2-996c-b1d052ba47f8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:15.274113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b41e52a-bd2f-44d2-996c-b1d052ba47f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:15.294943 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2b41e52a-bd2f-44d2-996c-b1d052ba47f8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:15.296266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2b41e52a-bd2f-44d2-996c-b1d052ba47f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:15.361734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1135/2296] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:15.371897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:15.373373 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:15.439502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bffa15e-f47c-4e09-a8c6-d399aede7481 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:15.443581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a996a2b6-5b2f-4484-a761-a08c52528883 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:15.444217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a996a2b6-5b2f-4484-a761-a08c52528883 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:15.444504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a996a2b6-5b2f-4484-a761-a08c52528883 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:15.444776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a996a2b6-5b2f-4484-a761-a08c52528883 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:15.455019 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a996a2b6-5b2f-4484-a761-a08c52528883 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:15.456581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a996a2b6-5b2f-4484-a761-a08c52528883 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:15.467389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1162/2297] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:15.781002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b41e52a-bd2f-44d2-996c-b1d052ba47f8 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:15.782531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2b41e52a-bd2f-44d2-996c-b1d052ba47f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:15.795776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b41e52a-bd2f-44d2-996c-b1d052ba47f8 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:15.801176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:15.801937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:15.802286 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:15.802630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] PATH_INFO: `/v3/projects/7eedce7d7b9b4d4ebb254e75c2971918` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:15.804045 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7eedce7d7b9b4d4ebb254e75c2971918)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:15.806435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:15.858691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7eedce7d7b9b4d4ebb254e75c2971918', 'request_id': 'req-26e65305-b29c-4bc2-a42b-00bfca50dd30'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.864547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7eedce7d7b9b4d4ebb254e75c2971918', 'request_id': 'req-26e65305-b29c-4bc2-a42b-00bfca50dd30'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.872122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7eedce7d7b9b4d4ebb254e75c2971918', 'request_id': 'req-26e65305-b29c-4bc2-a42b-00bfca50dd30'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.881061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:15.885206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] The token cache is being invalidate because project 7eedce7d7b9b4d4ebb254e75c2971918 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:15.885551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.886775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:15.890057 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1136/2298] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:15 2022] DELETE /identity/v3/projects/7eedce7d7b9b4d4ebb254e75c2971918 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:15.893433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a996a2b6-5b2f-4484-a761-a08c52528883 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:15.903130 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a996a2b6-5b2f-4484-a761-a08c52528883 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:15.957982 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:15.971193 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:15.972095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:15.972611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:15.972886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] PATH_INFO: `/v3/users/51a2347770114ead8b25b28bd858b7bd` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:15.974445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51a2347770114ead8b25b28bd858b7bd)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:15.977353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:16.041307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51a2347770114ead8b25b28bd858b7bd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.051201 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51a2347770114ead8b25b28bd858b7bd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.070881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51a2347770114ead8b25b28bd858b7bd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.071830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51a2347770114ead8b25b28bd858b7bd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.072701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51a2347770114ead8b25b28bd858b7bd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.092750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1163/2299] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:15 2022] DELETE /identity/v3/users/51a2347770114ead8b25b28bd858b7bd => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:16.466226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:16.468326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:16.485525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e65305-b29c-4bc2-a42b-00bfca50dd30 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:16.549161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:16.549802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:16.549977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:16.550254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] PATH_INFO: `/v3/projects/d3403583d2964fc0969d69ce53bb7b6a` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:16.551395 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3403583d2964fc0969d69ce53bb7b6a)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:16.553373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:16.602462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3403583d2964fc0969d69ce53bb7b6a', 'request_id': 'req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.609325 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3403583d2964fc0969d69ce53bb7b6a', 'request_id': 'req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.616384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3403583d2964fc0969d69ce53bb7b6a', 'request_id': 'req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.631293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] The token cache is being invalidate because project d3403583d2964fc0969d69ce53bb7b6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:16.631580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.632244 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.633767 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1137/2300] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:16 2022] DELETE /identity/v3/projects/d3403583d2964fc0969d69ce53bb7b6a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:16.638694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:16.640287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:16.653983 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537540e3-fbf8-4068-8507-16dbce3fe224 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:16.712367 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:16.712998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:16.713278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:16.713550 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] PATH_INFO: `/v3/users/b48ce49931ba40fd9d553aa9a12b4ddd` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:16.715224 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b48ce49931ba40fd9d553aa9a12b4ddd)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:16.717166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:16.769424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b48ce49931ba40fd9d553aa9a12b4ddd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.777588 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b48ce49931ba40fd9d553aa9a12b4ddd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.790656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b48ce49931ba40fd9d553aa9a12b4ddd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.791439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b48ce49931ba40fd9d553aa9a12b4ddd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.792054 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0dacb96f-786c-4987-9833-e15a7f480a79 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b48ce49931ba40fd9d553aa9a12b4ddd'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:16.803227 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:16.819490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1164/2301] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:16 2022] DELETE /identity/v3/users/b48ce49931ba40fd9d553aa9a12b4ddd => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:17.516041 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:17.517524 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:17.531994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa521f1a-26b2-41f8-8ea5-1e8e981a8b0f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:17.591265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:17.591883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:17.592187 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:17.592463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] PATH_INFO: `/v3/projects/5448fb5300304d38b9734af07c3ba834` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:17.593575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5448fb5300304d38b9734af07c3ba834)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:17.595480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:17.639149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:17.653635 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5448fb5300304d38b9734af07c3ba834', 'request_id': 'req-b1accaca-1b71-4c48-9480-73a8abe6d93f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:17.659257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5448fb5300304d38b9734af07c3ba834', 'request_id': 'req-b1accaca-1b71-4c48-9480-73a8abe6d93f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:17.665934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5448fb5300304d38b9734af07c3ba834', 'request_id': 'req-b1accaca-1b71-4c48-9480-73a8abe6d93f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:17.680382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] The token cache is being invalidate because project 5448fb5300304d38b9734af07c3ba834 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:17.680667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:17.681291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b1accaca-1b71-4c48-9480-73a8abe6d93f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:17.682752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1138/2302] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:17 2022] DELETE /identity/v3/projects/5448fb5300304d38b9734af07c3ba834 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:17.688520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39cc70d-1ab5-4ff0-ba95-13e8449bc3f4 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:17.689032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39cc70d-1ab5-4ff0-ba95-13e8449bc3f4 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:17.689136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39cc70d-1ab5-4ff0-ba95-13e8449bc3f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:17.714133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-c39cc70d-1ab5-4ff0-ba95-13e8449bc3f4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:17.715441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c39cc70d-1ab5-4ff0-ba95-13e8449bc3f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:17.800645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1165/2303] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:17.811826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83aef81-c4f3-4913-874a-774c97ce57e9 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:17.811826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83aef81-c4f3-4913-874a-774c97ce57e9 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:17.812059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83aef81-c4f3-4913-874a-774c97ce57e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:17.838758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-a83aef81-c4f3-4913-874a-774c97ce57e9 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:17.838758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a83aef81-c4f3-4913-874a-774c97ce57e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:17.859384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1139/2304] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:17.863709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39cc70d-1ab5-4ff0-ba95-13e8449bc3f4 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:17.874685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39cc70d-1ab5-4ff0-ba95-13e8449bc3f4 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:17.879556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:17.880195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:17.880480 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:17.880755 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:17.881408 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:17.881712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:17.883296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:17.897712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1166/2305] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:17.901960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83aef81-c4f3-4913-874a-774c97ce57e9 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:17.912819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83aef81-c4f3-4913-874a-774c97ce57e9 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:17.918661 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bf18d7-5438-488c-ba1a-465abbde730b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:17.919665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bf18d7-5438-488c-ba1a-465abbde730b admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:17.919824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bf18d7-5438-488c-ba1a-465abbde730b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:17.920076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bf18d7-5438-488c-ba1a-465abbde730b admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:17.920753 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bf18d7-5438-488c-ba1a-465abbde730b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:17.922665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bf18d7-5438-488c-ba1a-465abbde730b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:17.952466 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1140/2306] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:17 2022] POST /identity/v3/projects => generated 391 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:17.963315 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:17.974197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed377a40-461a-4d64-a11e-f5db956ffe15 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:18.033104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0fac26-61a7-4b73-8770-c70523476114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:18.033747 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0fac26-61a7-4b73-8770-c70523476114 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:18.034030 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0fac26-61a7-4b73-8770-c70523476114 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:18.034305 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0fac26-61a7-4b73-8770-c70523476114 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:18.034955 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0fac26-61a7-4b73-8770-c70523476114 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:18.036966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0fac26-61a7-4b73-8770-c70523476114 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:18.062355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1167/2307] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:17 2022] POST /identity/v3/users => generated 296 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:18.066726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bf18d7-5438-488c-ba1a-465abbde730b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:18.077874 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bf18d7-5438-488c-ba1a-465abbde730b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:18.082540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17d49cf-230e-4705-9f25-835b2b746c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:18.083147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17d49cf-230e-4705-9f25-835b2b746c8d admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:18.083416 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17d49cf-230e-4705-9f25-835b2b746c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:18.083700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17d49cf-230e-4705-9f25-835b2b746c8d admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:18.084407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17d49cf-230e-4705-9f25-835b2b746c8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:18.086046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17d49cf-230e-4705-9f25-835b2b746c8d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:18.097239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1141/2308] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:18.101433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0fac26-61a7-4b73-8770-c70523476114 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:18.112332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0fac26-61a7-4b73-8770-c70523476114 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:18.118230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76c5bb7-6bcc-4b8f-a364-7ba1438da506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:18.118840 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76c5bb7-6bcc-4b8f-a364-7ba1438da506 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:18.119122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76c5bb7-6bcc-4b8f-a364-7ba1438da506 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:18.119455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76c5bb7-6bcc-4b8f-a364-7ba1438da506 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:18.120115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76c5bb7-6bcc-4b8f-a364-7ba1438da506 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:18.122530 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76c5bb7-6bcc-4b8f-a364-7ba1438da506 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:18.133831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1168/2309] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:18.138158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17d49cf-230e-4705-9f25-835b2b746c8d admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:18.148779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17d49cf-230e-4705-9f25-835b2b746c8d admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:18.153780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:18.154464 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:18.154745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:18.155013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] PATH_INFO: `/v3/projects/9b5f5cc5e1684a8abe3ec54303fb8722/users/de865f4d44d94a27861f396308af21af/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:18.171833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b5f5cc5e1684a8abe3ec54303fb8722, user_id=de865f4d44d94a27861f396308af21af, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:18.180225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:18.198790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1142/2310] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:18 2022] PUT /identity/v3/projects/9b5f5cc5e1684a8abe3ec54303fb8722/users/de865f4d44d94a27861f396308af21af/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:18.210847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42840bbd-c99f-47c8-ab97-216f0c180db3 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:18.211160 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42840bbd-c99f-47c8-ab97-216f0c180db3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:18.211434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42840bbd-c99f-47c8-ab97-216f0c180db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:18.244961 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-42840bbd-c99f-47c8-ab97-216f0c180db3 None None] MFA Rules not processed for user `de865f4d44d94a27861f396308af21af`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:18.246863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-42840bbd-c99f-47c8-ab97-216f0c180db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:18.304487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:18.327218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1169/2311] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:18 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:18.338479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:18.341196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:18.398239 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc840b5e-3cf6-44d8-b5ca-ad616d2f6852 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:18.402929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81ad3e3-42af-4be2-b1f3-4d31428124eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:18.403551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81ad3e3-42af-4be2-b1f3-4d31428124eb service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:18.403832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81ad3e3-42af-4be2-b1f3-4d31428124eb service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:18.404103 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81ad3e3-42af-4be2-b1f3-4d31428124eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:18.412955 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81ad3e3-42af-4be2-b1f3-4d31428124eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:18.414394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81ad3e3-42af-4be2-b1f3-4d31428124eb service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:18.424547 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1143/2312] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:18 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:20.817244 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42840bbd-c99f-47c8-ab97-216f0c180db3 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:20.818667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-42840bbd-c99f-47c8-ab97-216f0c180db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:20.884109 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42840bbd-c99f-47c8-ab97-216f0c180db3 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:20.888908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:20.889526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:20.889807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:20.890082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] PATH_INFO: `/v3/users/5e063f7f37ae43cdb2016f3bab247184` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:20.891197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e063f7f37ae43cdb2016f3bab247184)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:20.893157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:20.951502 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e063f7f37ae43cdb2016f3bab247184'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:20.959010 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e063f7f37ae43cdb2016f3bab247184'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:20.981404 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e063f7f37ae43cdb2016f3bab247184'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:20.982892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e063f7f37ae43cdb2016f3bab247184'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:20.983525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e063f7f37ae43cdb2016f3bab247184'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:21.004621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1170/2313] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:20 2022] DELETE /identity/v3/users/5e063f7f37ae43cdb2016f3bab247184 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:21.009708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d241bd61-fad0-4753-96c7-39e379f34e33 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.010512 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d241bd61-fad0-4753-96c7-39e379f34e33 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.011009 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d241bd61-fad0-4753-96c7-39e379f34e33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.033597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-d241bd61-fad0-4753-96c7-39e379f34e33 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:21.035362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d241bd61-fad0-4753-96c7-39e379f34e33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:21.101646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:21.120362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1144/2314] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:21.127448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:21.129136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:21.201064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720ff2c5-de73-4625-b78d-177cde87eb35 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:21.204645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:21.205217 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.205501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.205769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.216438 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:21.217868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:21.228245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1171/2315] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:21.461968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d241bd61-fad0-4753-96c7-39e379f34e33 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:21.463757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-d241bd61-fad0-4753-96c7-39e379f34e33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:21.550610 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:21.552460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:21.567823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d241bd61-fad0-4753-96c7-39e379f34e33 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:21.573956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecdfef9-930c-4850-9519-1d458ba38b37 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:21.574404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6138f9-dd4c-4632-91bd-a34ed2b415c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:21.575170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6138f9-dd4c-4632-91bd-a34ed2b415c9 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.575513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6138f9-dd4c-4632-91bd-a34ed2b415c9 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.575888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6138f9-dd4c-4632-91bd-a34ed2b415c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.577763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3b6138f9-dd4c-4632-91bd-a34ed2b415c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:21.579424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:21.580056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.580374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.580651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] PATH_INFO: `/v3/projects/51f1284fd166460bbd7243217b6da7d1` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.582294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] RBAC: Authorizing `identity:delete_project(project_id=51f1284fd166460bbd7243217b6da7d1)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:21.585164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:21.588564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:21.600865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6138f9-dd4c-4632-91bd-a34ed2b415c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:21.602718 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6138f9-dd4c-4632-91bd-a34ed2b415c9 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:21.618069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1145/2316] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:21.660449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51f1284fd166460bbd7243217b6da7d1', 'request_id': 'req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:21.667092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51f1284fd166460bbd7243217b6da7d1', 'request_id': 'req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:21.675836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51f1284fd166460bbd7243217b6da7d1', 'request_id': 'req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:21.687927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] The token cache is being invalidate because project 51f1284fd166460bbd7243217b6da7d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:21.688246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:21.689027 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-593e0522-f3fe-4d1e-9db1-2c2c06a3d126 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:21.690698 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1172/2317] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:21 2022] DELETE /identity/v3/projects/51f1284fd166460bbd7243217b6da7d1 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:21.695616 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68168e-5499-4fc1-8154-9096b2c8be8f None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.695897 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68168e-5499-4fc1-8154-9096b2c8be8f None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.696193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68168e-5499-4fc1-8154-9096b2c8be8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.718255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9e68168e-5499-4fc1-8154-9096b2c8be8f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:21.719924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9e68168e-5499-4fc1-8154-9096b2c8be8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:21.784777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1146/2318] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:21.792624 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137b3730-3b1a-4cfd-9a49-c2d6f2016c04 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.793037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137b3730-3b1a-4cfd-9a49-c2d6f2016c04 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.793333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137b3730-3b1a-4cfd-9a49-c2d6f2016c04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.815752 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-137b3730-3b1a-4cfd-9a49-c2d6f2016c04 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:21.817024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-137b3730-3b1a-4cfd-9a49-c2d6f2016c04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:21.825406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1173/2319] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:21.829653 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e68168e-5499-4fc1-8154-9096b2c8be8f None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:21.840193 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e68168e-5499-4fc1-8154-9096b2c8be8f None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:21.845033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:21.845626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.845907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.846207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.846851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:21.847158 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:21.848797 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:21.860452 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1147/2320] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:21.864855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137b3730-3b1a-4cfd-9a49-c2d6f2016c04 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:21.874519 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137b3730-3b1a-4cfd-9a49-c2d6f2016c04 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:21.879160 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1bd42e-469f-4f2e-985e-124cab2b4d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:21.879755 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1bd42e-469f-4f2e-985e-124cab2b4d49 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.880159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1bd42e-469f-4f2e-985e-124cab2b4d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.880445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1bd42e-469f-4f2e-985e-124cab2b4d49 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.881187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1bd42e-469f-4f2e-985e-124cab2b4d49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:21.883090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1bd42e-469f-4f2e-985e-124cab2b4d49 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:21.913086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1174/2321] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:21 2022] POST /identity/v3/projects => generated 403 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:21.915969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:21.926227 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07253ceb-c2ea-44d0-92dd-d6b93c925597 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:21.984834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119902f-40ff-4c4c-8c3c-74d4b02fe6ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:21.985860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119902f-40ff-4c4c-8c3c-74d4b02fe6ee admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:21.985860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119902f-40ff-4c4c-8c3c-74d4b02fe6ee admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:21.985860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119902f-40ff-4c4c-8c3c-74d4b02fe6ee admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:21.986499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119902f-40ff-4c4c-8c3c-74d4b02fe6ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:21.988462 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119902f-40ff-4c4c-8c3c-74d4b02fe6ee admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:22.010798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1148/2322] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:21 2022] POST /identity/v3/users => generated 302 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:22.018286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1bd42e-469f-4f2e-985e-124cab2b4d49 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:22.028922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1bd42e-469f-4f2e-985e-124cab2b4d49 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:22.035202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f49b5f-ce41-4d30-9b77-87b16b62df68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:22.035814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f49b5f-ce41-4d30-9b77-87b16b62df68 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:22.036096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f49b5f-ce41-4d30-9b77-87b16b62df68 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:22.036413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f49b5f-ce41-4d30-9b77-87b16b62df68 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:22.037060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f49b5f-ce41-4d30-9b77-87b16b62df68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:22.038684 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f49b5f-ce41-4d30-9b77-87b16b62df68 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:22.049464 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1175/2323] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:22.053614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119902f-40ff-4c4c-8c3c-74d4b02fe6ee admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:22.059998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:22.066735 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119902f-40ff-4c4c-8c3c-74d4b02fe6ee admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:22.072150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9462c3ba-c825-4e49-8077-d41e7209517c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:22.072752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462c3ba-c825-4e49-8077-d41e7209517c admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:22.073028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462c3ba-c825-4e49-8077-d41e7209517c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:22.073299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462c3ba-c825-4e49-8077-d41e7209517c admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:22.075248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9462c3ba-c825-4e49-8077-d41e7209517c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:22.076905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9462c3ba-c825-4e49-8077-d41e7209517c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:22.089308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1149/2324] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:22.094099 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f49b5f-ce41-4d30-9b77-87b16b62df68 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:22.104411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f49b5f-ce41-4d30-9b77-87b16b62df68 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:22.109718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:22.110367 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:22.110649 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:22.110925 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] PATH_INFO: `/v3/projects/cc5ff0d3913047af84aba630f051ac51/users/c9ee3fc86df14e09b38f6ca1cfd31ba8/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:22.131593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc5ff0d3913047af84aba630f051ac51, user_id=c9ee3fc86df14e09b38f6ca1cfd31ba8, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:22.134368 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:22.150401 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1176/2325] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:22 2022] PUT /identity/v3/projects/cc5ff0d3913047af84aba630f051ac51/users/c9ee3fc86df14e09b38f6ca1cfd31ba8/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:22.155885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6dc7a8-b08c-4b64-b071-5cf2600c71ec None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:22.156342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6dc7a8-b08c-4b64-b071-5cf2600c71ec None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:22.156637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6dc7a8-b08c-4b64-b071-5cf2600c71ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:22.177854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7b6dc7a8-b08c-4b64-b071-5cf2600c71ec None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:22.179093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7b6dc7a8-b08c-4b64-b071-5cf2600c71ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:22.249326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1150/2326] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:22.259565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:22.262055 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:22.325369 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d9a443-0981-4a3c-affa-c6e85f24d18d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:22.329100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296fbc23-ba58-4a9e-be30-942be1d32351 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:22.329707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296fbc23-ba58-4a9e-be30-942be1d32351 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:22.329990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296fbc23-ba58-4a9e-be30-942be1d32351 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:22.330266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296fbc23-ba58-4a9e-be30-942be1d32351 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:22.339358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296fbc23-ba58-4a9e-be30-942be1d32351 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:22.340819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296fbc23-ba58-4a9e-be30-942be1d32351 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:22.350632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1177/2327] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:22.985456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605305a-1185-4d73-8cee-e3fb743601e8 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:22.985711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605305a-1185-4d73-8cee-e3fb743601e8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:22.986050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605305a-1185-4d73-8cee-e3fb743601e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:23.020326 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-5605305a-1185-4d73-8cee-e3fb743601e8 None None] MFA Rules not processed for user `c9ee3fc86df14e09b38f6ca1cfd31ba8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:23.021551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5605305a-1185-4d73-8cee-e3fb743601e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:23.104405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1151/2328] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:22 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:23.109573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296fbc23-ba58-4a9e-be30-942be1d32351 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:23.121578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296fbc23-ba58-4a9e-be30-942be1d32351 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:23.126206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84179c-bb45-4d01-979d-29859043d983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:23.126811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84179c-bb45-4d01-979d-29859043d983 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:23.127090 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84179c-bb45-4d01-979d-29859043d983 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:23.127363 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84179c-bb45-4d01-979d-29859043d983 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:23.128011 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a84179c-bb45-4d01-979d-29859043d983 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:23.130084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a84179c-bb45-4d01-979d-29859043d983 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:23.164207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1178/2329] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:23 2022] POST /identity/v3/projects => generated 403 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:23.165461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5605305a-1185-4d73-8cee-e3fb743601e8 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:23.178848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5605305a-1185-4d73-8cee-e3fb743601e8 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:23.252333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19528f5-51c7-44b9-b535-ffa4d689cb44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:23.252956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19528f5-51c7-44b9-b535-ffa4d689cb44 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:23.253237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19528f5-51c7-44b9-b535-ffa4d689cb44 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:23.253510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19528f5-51c7-44b9-b535-ffa4d689cb44 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:23.254150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19528f5-51c7-44b9-b535-ffa4d689cb44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:23.256087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19528f5-51c7-44b9-b535-ffa4d689cb44 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:23.301126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1152/2330] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:23 2022] POST /identity/v3/users => generated 308 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:23.311407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84179c-bb45-4d01-979d-29859043d983 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:23.322231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84179c-bb45-4d01-979d-29859043d983 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:23.327280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356144f-9aa5-43d0-85ca-667dbdb29b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:23.327995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356144f-9aa5-43d0-85ca-667dbdb29b2a admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:23.328400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356144f-9aa5-43d0-85ca-667dbdb29b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:23.328769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356144f-9aa5-43d0-85ca-667dbdb29b2a admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:23.329501 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9356144f-9aa5-43d0-85ca-667dbdb29b2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:23.331931 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9356144f-9aa5-43d0-85ca-667dbdb29b2a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:23.334455 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:23.348100 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1179/2331] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:23.349368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19528f5-51c7-44b9-b535-ffa4d689cb44 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:23.360913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19528f5-51c7-44b9-b535-ffa4d689cb44 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:23.365752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c393e4ff-5567-4ff9-bae3-a640f59f2065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:23.366666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c393e4ff-5567-4ff9-bae3-a640f59f2065 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:23.366950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c393e4ff-5567-4ff9-bae3-a640f59f2065 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:23.367229 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c393e4ff-5567-4ff9-bae3-a640f59f2065 admin admin] PATH_INFO: `/v3/projects/c89d26e727a247d8af9a7a0cf89c2692/users/4c8c82e06f5e46eeb7ce441254e79808/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:23.376896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:23.385314 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c393e4ff-5567-4ff9-bae3-a640f59f2065 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c89d26e727a247d8af9a7a0cf89c2692, user_id=4c8c82e06f5e46eeb7ce441254e79808, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:23.388481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c393e4ff-5567-4ff9-bae3-a640f59f2065 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:23.401850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1153/2332] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:23 2022] PUT /identity/v3/projects/c89d26e727a247d8af9a7a0cf89c2692/users/4c8c82e06f5e46eeb7ce441254e79808/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:23.407748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356144f-9aa5-43d0-85ca-667dbdb29b2a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:23.419065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356144f-9aa5-43d0-85ca-667dbdb29b2a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:23.478198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a985c7b1-5501-46e8-999a-255c3b6949a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:23.478832 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a985c7b1-5501-46e8-999a-255c3b6949a9 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:23.479107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a985c7b1-5501-46e8-999a-255c3b6949a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:23.479377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a985c7b1-5501-46e8-999a-255c3b6949a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:23.480628 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a985c7b1-5501-46e8-999a-255c3b6949a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:23.482859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a985c7b1-5501-46e8-999a-255c3b6949a9 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:23.493725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1180/2333] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:23.499006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c393e4ff-5567-4ff9-bae3-a640f59f2065 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:23.509186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c393e4ff-5567-4ff9-bae3-a640f59f2065 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:23.514335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f2882-f3f5-423a-b884-efe7a415b963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:23.514967 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f2882-f3f5-423a-b884-efe7a415b963 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:23.515240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f2882-f3f5-423a-b884-efe7a415b963 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:23.515507 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f2882-f3f5-423a-b884-efe7a415b963 admin admin] PATH_INFO: `/v3/projects/c89d26e727a247d8af9a7a0cf89c2692/users/4c8c82e06f5e46eeb7ce441254e79808/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:23.517407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341f2882-f3f5-423a-b884-efe7a415b963 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c89d26e727a247d8af9a7a0cf89c2692, user_id=4c8c82e06f5e46eeb7ce441254e79808, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:23.520025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341f2882-f3f5-423a-b884-efe7a415b963 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:23.531242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1154/2334] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:23 2022] PUT /identity/v3/projects/c89d26e727a247d8af9a7a0cf89c2692/users/4c8c82e06f5e46eeb7ce441254e79808/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:24.199328 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4398e1e-d1fe-4602-8bc7-9f8240e1f8b2 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:24.199915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4398e1e-d1fe-4602-8bc7-9f8240e1f8b2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:24.199915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4398e1e-d1fe-4602-8bc7-9f8240e1f8b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:24.235219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d4398e1e-d1fe-4602-8bc7-9f8240e1f8b2 None None] MFA Rules not processed for user `4c8c82e06f5e46eeb7ce441254e79808`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:24.236596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d4398e1e-d1fe-4602-8bc7-9f8240e1f8b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:24.313057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1181/2335] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:24 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:25.556303 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f2882-f3f5-423a-b884-efe7a415b963 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:25.629371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f2882-f3f5-423a-b884-efe7a415b963 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:25.634095 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:25.634721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:25.635026 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:25.635306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] PATH_INFO: `/v3/users/c9ee3fc86df14e09b38f6ca1cfd31ba8` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:25.636647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9ee3fc86df14e09b38f6ca1cfd31ba8)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:25.638575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:25.698211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9ee3fc86df14e09b38f6ca1cfd31ba8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:25.705408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9ee3fc86df14e09b38f6ca1cfd31ba8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:25.718672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9ee3fc86df14e09b38f6ca1cfd31ba8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:25.719408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9ee3fc86df14e09b38f6ca1cfd31ba8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:25.720056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9ee3fc86df14e09b38f6ca1cfd31ba8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:25.735512 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1155/2336] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:25 2022] DELETE /identity/v3/users/c9ee3fc86df14e09b38f6ca1cfd31ba8 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:25.932487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4398e1e-d1fe-4602-8bc7-9f8240e1f8b2 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:25.943407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4398e1e-d1fe-4602-8bc7-9f8240e1f8b2 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.007189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.007950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.008309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.008692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] PATH_INFO: `/v3/projects/cc5ff0d3913047af84aba630f051ac51` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.010008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc5ff0d3913047af84aba630f051ac51)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.012008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:26.035223 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:26.060213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc5ff0d3913047af84aba630f051ac51', 'request_id': 'req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.066858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc5ff0d3913047af84aba630f051ac51', 'request_id': 'req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.075421 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc5ff0d3913047af84aba630f051ac51', 'request_id': 'req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.087763 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] The token cache is being invalidate because project cc5ff0d3913047af84aba630f051ac51 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:26.088166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.088844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.091104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1182/2337] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:25 2022] DELETE /identity/v3/projects/cc5ff0d3913047af84aba630f051ac51 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:26.094931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:26.096461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:26.110831 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28663d50-6876-402e-91ea-ebb2b209efe5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.172685 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:26.184347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.186078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.186365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.186646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] PATH_INFO: `/v3/users/4c8c82e06f5e46eeb7ce441254e79808` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.187834 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c8c82e06f5e46eeb7ce441254e79808)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.189902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:26.232633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:26.234015 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:26.245175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c8c82e06f5e46eeb7ce441254e79808'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.247932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abf0e20-b05a-4e4d-9e8e-fd090361eff4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.252683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.252843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c8c82e06f5e46eeb7ce441254e79808'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.253288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.253568 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.253846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] PATH_INFO: `/v3/users/caed935a15c34fd888ce2974aa28839b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.255012 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] RBAC: Authorizing `identity:delete_user(user_id=caed935a15c34fd888ce2974aa28839b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.256967 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:26.266125 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c8c82e06f5e46eeb7ce441254e79808'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.266921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c8c82e06f5e46eeb7ce441254e79808'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.267560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-228378e6-947b-48c2-9fd2-ede164da5d06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c8c82e06f5e46eeb7ce441254e79808'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.283280 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1156/2338] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:26 2022] DELETE /identity/v3/users/4c8c82e06f5e46eeb7ce441254e79808 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:26.312970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'caed935a15c34fd888ce2974aa28839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.319932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'caed935a15c34fd888ce2974aa28839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.332947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'caed935a15c34fd888ce2974aa28839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.333680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'caed935a15c34fd888ce2974aa28839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.334280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'caed935a15c34fd888ce2974aa28839b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.347985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1183/2339] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:26 2022] DELETE /identity/v3/users/caed935a15c34fd888ce2974aa28839b => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:26.352472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d0a6c6-9c51-4f19-b301-d29d61c317f4 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.352760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d0a6c6-9c51-4f19-b301-d29d61c317f4 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.353016 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d0a6c6-9c51-4f19-b301-d29d61c317f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.375049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-01d0a6c6-9c51-4f19-b301-d29d61c317f4 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:26.376284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-01d0a6c6-9c51-4f19-b301-d29d61c317f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:26.443531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1157/2340] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:26.456044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:26.457520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:26.477194 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d0a6c6-9c51-4f19-b301-d29d61c317f4 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:26.478777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-01d0a6c6-9c51-4f19-b301-d29d61c317f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:26.493512 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d0a6c6-9c51-4f19-b301-d29d61c317f4 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.502815 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.503475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.503759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.504034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] PATH_INFO: `/v3/projects/c89d26e727a247d8af9a7a0cf89c2692` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.505276 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] RBAC: Authorizing `identity:delete_project(project_id=c89d26e727a247d8af9a7a0cf89c2692)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.507165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:26.555116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38dc4f8-0a45-482f-9f26-48f4bb21438d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.569871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c89d26e727a247d8af9a7a0cf89c2692', 'request_id': 'req-b599a430-bc82-4804-a41e-e2bd391c2efa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.577750 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c89d26e727a247d8af9a7a0cf89c2692', 'request_id': 'req-b599a430-bc82-4804-a41e-e2bd391c2efa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.584987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c89d26e727a247d8af9a7a0cf89c2692', 'request_id': 'req-b599a430-bc82-4804-a41e-e2bd391c2efa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.588947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:26.598706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:26.600709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] The token cache is being invalidate because project c89d26e727a247d8af9a7a0cf89c2692 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:26.601240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.602608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b599a430-bc82-4804-a41e-e2bd391c2efa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:26.604872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1158/2341] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:26 2022] DELETE /identity/v3/projects/c89d26e727a247d8af9a7a0cf89c2692 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:26.610459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd6967-0d3c-4450-bb88-c5a8d55aa1f3 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.610769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd6967-0d3c-4450-bb88-c5a8d55aa1f3 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.611085 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd6967-0d3c-4450-bb88-c5a8d55aa1f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.620105 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.621558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.622270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.622270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.623731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:26.633680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-83dd6967-0d3c-4450-bb88-c5a8d55aa1f3 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:26.634908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-83dd6967-0d3c-4450-bb88-c5a8d55aa1f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:26.694044 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.695558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:26.703067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1159/2342] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:26.706696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1184/2343] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:26.710770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7fe82d-9d26-4b31-b99c-e7391aeec1a2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.711058 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7fe82d-9d26-4b31-b99c-e7391aeec1a2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.711334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7fe82d-9d26-4b31-b99c-e7391aeec1a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.733780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-da7fe82d-9d26-4b31-b99c-e7391aeec1a2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:26.735025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-da7fe82d-9d26-4b31-b99c-e7391aeec1a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:26.743917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1160/2344] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:26.749290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:26.759695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00f9516-00a4-4647-8074-7a1dce3b0c0e service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.766491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.767103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.767386 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.767680 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.768617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:26.768929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.771294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:26.786389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1185/2345] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:26.793670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7fe82d-9d26-4b31-b99c-e7391aeec1a2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:26.804612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7fe82d-9d26-4b31-b99c-e7391aeec1a2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.810731 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b864ee0-f139-4a63-9b97-3ac15c1c2271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.811426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b864ee0-f139-4a63-9b97-3ac15c1c2271 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.811724 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b864ee0-f139-4a63-9b97-3ac15c1c2271 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.812025 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b864ee0-f139-4a63-9b97-3ac15c1c2271 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.812823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b864ee0-f139-4a63-9b97-3ac15c1c2271 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.815116 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b864ee0-f139-4a63-9b97-3ac15c1c2271 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:26.847801 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1161/2346] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:26 2022] POST /identity/v3/projects => generated 365 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:26.849139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:26.862427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fedbcd-0c87-4220-86a8-5e3a3025831d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.924356 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c70e4-1e95-44b9-b291-cd8bfc2379c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.925697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c70e4-1e95-44b9-b291-cd8bfc2379c7 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.925697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c70e4-1e95-44b9-b291-cd8bfc2379c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.926169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c70e4-1e95-44b9-b291-cd8bfc2379c7 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.926911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4c70e4-1e95-44b9-b291-cd8bfc2379c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.929035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4c70e4-1e95-44b9-b291-cd8bfc2379c7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:26.954603 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1186/2347] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:26 2022] POST /identity/v3/users => generated 283 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:26.958382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b864ee0-f139-4a63-9b97-3ac15c1c2271 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:26.971660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b864ee0-f139-4a63-9b97-3ac15c1c2271 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:26.983619 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:26.984433 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:26.984982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:26.985361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:26.991481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:26.991481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.010122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c70e4-1e95-44b9-b291-cd8bfc2379c7 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.020576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1162/2348] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:26 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:27.022130 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.024823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.026572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c70e4-1e95-44b9-b291-cd8bfc2379c7 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.033661 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5914d720-33cc-4a5c-a8c0-c7aeec28c81f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.034354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5914d720-33cc-4a5c-a8c0-c7aeec28c81f admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.034689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5914d720-33cc-4a5c-a8c0-c7aeec28c81f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.034978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5914d720-33cc-4a5c-a8c0-c7aeec28c81f admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.035601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5914d720-33cc-4a5c-a8c0-c7aeec28c81f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.037250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5914d720-33cc-4a5c-a8c0-c7aeec28c81f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.050796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1187/2349] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:27 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:27.055181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5914d720-33cc-4a5c-a8c0-c7aeec28c81f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.058780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5dc1eff-e7d6-479b-b2fa-bb72e65752f1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.066760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.067390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.067672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.067941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] PATH_INFO: `/v3/projects/082e59576f304acb9858b1fa2605344d` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.068067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5914d720-33cc-4a5c-a8c0-c7aeec28c81f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.069353 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] RBAC: Authorizing `identity:delete_project(project_id=082e59576f304acb9858b1fa2605344d)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.071342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.073170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a6881e-555a-4538-9321-2f196311926d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.073878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a6881e-555a-4538-9321-2f196311926d admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.074259 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a6881e-555a-4538-9321-2f196311926d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.074629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a6881e-555a-4538-9321-2f196311926d admin admin] PATH_INFO: `/v3/projects/44659d3fd80449a6b525672c1e4b5195/users/eacd53e5a96447259ee6c4877c53e784/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.096081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a6881e-555a-4538-9321-2f196311926d admin admin] RBAC: Authorizing `identity:create_grant(project_id=44659d3fd80449a6b525672c1e4b5195, user_id=eacd53e5a96447259ee6c4877c53e784, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.098731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a6881e-555a-4538-9321-2f196311926d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.115123 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1188/2350] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:27 2022] PUT /identity/v3/projects/44659d3fd80449a6b525672c1e4b5195/users/eacd53e5a96447259ee6c4877c53e784/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:27.124445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '082e59576f304acb9858b1fa2605344d', 'request_id': 'req-ee8a4846-e2d4-42d7-b5d0-91871debc43c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.127771 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac97077e-2c35-4917-a9e7-448edcba13b0 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.128025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac97077e-2c35-4917-a9e7-448edcba13b0 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.128345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac97077e-2c35-4917-a9e7-448edcba13b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.130589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '082e59576f304acb9858b1fa2605344d', 'request_id': 'req-ee8a4846-e2d4-42d7-b5d0-91871debc43c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.140943 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '082e59576f304acb9858b1fa2605344d', 'request_id': 'req-ee8a4846-e2d4-42d7-b5d0-91871debc43c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.154091 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] The token cache is being invalidate because project 082e59576f304acb9858b1fa2605344d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:27.154384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.155098 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.156697 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1163/2351] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:27 2022] DELETE /identity/v3/projects/082e59576f304acb9858b1fa2605344d => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:27.160548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.161298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-ac97077e-2c35-4917-a9e7-448edcba13b0 None None] MFA Rules not processed for user `eacd53e5a96447259ee6c4877c53e784`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:27.161935 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.162466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ac97077e-2c35-4917-a9e7-448edcba13b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.181887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:27.208707 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:27.235184 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1189/2352] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:27 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:27.235880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a4846-e2d4-42d7-b5d0-91871debc43c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.239807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac97077e-2c35-4917-a9e7-448edcba13b0 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.242000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.242230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ac97077e-2c35-4917-a9e7-448edcba13b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.242629 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.242905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.243174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] PATH_INFO: `/v3/users/2e79cf35ad454c63aa593ad1782ba73a` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.244343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e79cf35ad454c63aa593ad1782ba73a)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.247549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.256298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac97077e-2c35-4917-a9e7-448edcba13b0 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.261162 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c954e46-cd78-4a0c-8e6b-8b839aad68fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.261872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c954e46-cd78-4a0c-8e6b-8b839aad68fa admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.262342 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c954e46-cd78-4a0c-8e6b-8b839aad68fa admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.262611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c954e46-cd78-4a0c-8e6b-8b839aad68fa admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.263240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c954e46-cd78-4a0c-8e6b-8b839aad68fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.265216 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c954e46-cd78-4a0c-8e6b-8b839aad68fa admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.296183 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1190/2353] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:27 2022] POST /identity/v3/projects => generated 365 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:27.296896 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e79cf35ad454c63aa593ad1782ba73a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.299096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c954e46-cd78-4a0c-8e6b-8b839aad68fa admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.306124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e79cf35ad454c63aa593ad1782ba73a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.319936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e79cf35ad454c63aa593ad1782ba73a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.320719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e79cf35ad454c63aa593ad1782ba73a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.321344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e79cf35ad454c63aa593ad1782ba73a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.335484 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1164/2354] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:27 2022] DELETE /identity/v3/users/2e79cf35ad454c63aa593ad1782ba73a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:27.364492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c954e46-cd78-4a0c-8e6b-8b839aad68fa admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.426007 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949fe75-b62f-41a4-8b2e-5b82f0dc8e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.426841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949fe75-b62f-41a4-8b2e-5b82f0dc8e32 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.427266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949fe75-b62f-41a4-8b2e-5b82f0dc8e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.427713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949fe75-b62f-41a4-8b2e-5b82f0dc8e32 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.428407 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4949fe75-b62f-41a4-8b2e-5b82f0dc8e32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.430362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4949fe75-b62f-41a4-8b2e-5b82f0dc8e32 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.457001 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1191/2355] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:27 2022] POST /identity/v3/users => generated 289 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:27.458175 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.459670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.472922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c61f7eb-844f-425c-847b-ccf60def3d49 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.480523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d8a93-44c2-4cf8-9857-f8ed0d1b3734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.481134 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d8a93-44c2-4cf8-9857-f8ed0d1b3734 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.481419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d8a93-44c2-4cf8-9857-f8ed0d1b3734 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.481696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d8a93-44c2-4cf8-9857-f8ed0d1b3734 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.482349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d8a93-44c2-4cf8-9857-f8ed0d1b3734 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.484101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d8a93-44c2-4cf8-9857-f8ed0d1b3734 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.496490 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1165/2356] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:27 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:27.501326 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949fe75-b62f-41a4-8b2e-5b82f0dc8e32 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.512666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949fe75-b62f-41a4-8b2e-5b82f0dc8e32 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.522830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.523512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.523798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.524076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] PATH_INFO: `/v3/projects/6908414ecd914e6b83f63e14add2be34/users/c9d90f58b1ab4b84a0586b7d8611503d/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.541003 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6908414ecd914e6b83f63e14add2be34, user_id=c9d90f58b1ab4b84a0586b7d8611503d, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.544039 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.558793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1192/2357] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:27 2022] PUT /identity/v3/projects/6908414ecd914e6b83f63e14add2be34/users/c9d90f58b1ab4b84a0586b7d8611503d/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:27.563261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d8a93-44c2-4cf8-9857-f8ed0d1b3734 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.573657 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d8a93-44c2-4cf8-9857-f8ed0d1b3734 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.641038 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f2ead8-82df-45b2-a146-db4ebe832930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.641645 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f2ead8-82df-45b2-a146-db4ebe832930 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.641937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f2ead8-82df-45b2-a146-db4ebe832930 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.642180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f2ead8-82df-45b2-a146-db4ebe832930 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.645563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f2ead8-82df-45b2-a146-db4ebe832930 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.645563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f2ead8-82df-45b2-a146-db4ebe832930 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.647731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.649575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.659858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1166/2358] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:27 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:27.665071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f2ead8-82df-45b2-a146-db4ebe832930 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.668246 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1ebf6e-88c7-4d3a-8914-c78d5e88f4f3 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.679249 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f2ead8-82df-45b2-a146-db4ebe832930 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.681741 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.682393 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.682685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.682961 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] PATH_INFO: `/v3/projects/6bd809b216bd4f6994af711fbd30dcec` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.685002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bd809b216bd4f6994af711fbd30dcec)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.685343 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383adb5e-d438-4665-8b85-16effaa5b7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.685778 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383adb5e-d438-4665-8b85-16effaa5b7e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.686057 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383adb5e-d438-4665-8b85-16effaa5b7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.686324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383adb5e-d438-4665-8b85-16effaa5b7e2 admin admin] PATH_INFO: `/v3/projects/6908414ecd914e6b83f63e14add2be34/users/c9d90f58b1ab4b84a0586b7d8611503d/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.687102 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.688338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383adb5e-d438-4665-8b85-16effaa5b7e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6908414ecd914e6b83f63e14add2be34, user_id=c9d90f58b1ab4b84a0586b7d8611503d, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.690520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:27.690982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383adb5e-d438-4665-8b85-16effaa5b7e2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.707381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1167/2359] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:27 2022] PUT /identity/v3/projects/6908414ecd914e6b83f63e14add2be34/users/c9d90f58b1ab4b84a0586b7d8611503d/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:27.718993 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115ed5e-2e14-455c-9ba8-e03f37681e44 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.719277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115ed5e-2e14-455c-9ba8-e03f37681e44 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.719566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115ed5e-2e14-455c-9ba8-e03f37681e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.744645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bd809b216bd4f6994af711fbd30dcec', 'request_id': 'req-43be5e46-b49a-45d7-95f6-42c7beac51c2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.751996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-6115ed5e-2e14-455c-9ba8-e03f37681e44 None None] MFA Rules not processed for user `c9d90f58b1ab4b84a0586b7d8611503d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:27.752700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bd809b216bd4f6994af711fbd30dcec', 'request_id': 'req-43be5e46-b49a-45d7-95f6-42c7beac51c2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.753646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6115ed5e-2e14-455c-9ba8-e03f37681e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.762181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bd809b216bd4f6994af711fbd30dcec', 'request_id': 'req-43be5e46-b49a-45d7-95f6-42c7beac51c2'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.774411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] The token cache is being invalidate because project 6bd809b216bd4f6994af711fbd30dcec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:27.774689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.775248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:27.775481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-43be5e46-b49a-45d7-95f6-42c7beac51c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:27.779197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1193/2360] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:27 2022] DELETE /identity/v3/projects/6bd809b216bd4f6994af711fbd30dcec => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:27.783940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2110082-b6e8-4b7a-9606-5ffb84ffff27 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.784266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2110082-b6e8-4b7a-9606-5ffb84ffff27 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.784541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2110082-b6e8-4b7a-9606-5ffb84ffff27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.805895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d2110082-b6e8-4b7a-9606-5ffb84ffff27 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:27.807119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d2110082-b6e8-4b7a-9606-5ffb84ffff27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.831964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1168/2361] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:27 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:27.844455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6115ed5e-2e14-455c-9ba8-e03f37681e44 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.845865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6115ed5e-2e14-455c-9ba8-e03f37681e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.877943 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1194/2362] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:27.885805 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261195e-fe54-47f5-a9d3-6f624dbf83de None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.886089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261195e-fe54-47f5-a9d3-6f624dbf83de None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.886878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261195e-fe54-47f5-a9d3-6f624dbf83de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.907602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4261195e-fe54-47f5-a9d3-6f624dbf83de None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:27.908864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4261195e-fe54-47f5-a9d3-6f624dbf83de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:27.908980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6115ed5e-2e14-455c-9ba8-e03f37681e44 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.912716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8faa5-06a1-45ca-9ba6-96a2492cfb3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.913371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8faa5-06a1-45ca-9ba6-96a2492cfb3e service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.913625 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8faa5-06a1-45ca-9ba6-96a2492cfb3e service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.913909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8faa5-06a1-45ca-9ba6-96a2492cfb3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.917720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1195/2363] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:27.922914 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4261195e-fe54-47f5-a9d3-6f624dbf83de None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.924561 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd8faa5-06a1-45ca-9ba6-96a2492cfb3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.925998 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd8faa5-06a1-45ca-9ba6-96a2492cfb3e service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.931859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4261195e-fe54-47f5-a9d3-6f624dbf83de None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.936810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1169/2364] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:27 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:27.937298 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.937897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.938172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.938439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.939080 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:27.939377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.940999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:27.954258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1196/2365] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 09:02:27.958541 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8faa5-06a1-45ca-9ba6-96a2492cfb3e service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:27.971593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8faa5-06a1-45ca-9ba6-96a2492cfb3e service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:27.978928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207f63f5-665d-4adf-ad8c-6b3db0174c17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:27.979568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207f63f5-665d-4adf-ad8c-6b3db0174c17 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:27.979949 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207f63f5-665d-4adf-ad8c-6b3db0174c17 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:27.980327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207f63f5-665d-4adf-ad8c-6b3db0174c17 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:27.981037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207f63f5-665d-4adf-ad8c-6b3db0174c17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:27.983290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207f63f5-665d-4adf-ad8c-6b3db0174c17 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.016473 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1170/2366] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:27 2022] POST /identity/v3/projects => generated 373 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:28.020572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.032441 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f44751-a651-470c-8034-790cafb9642a admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.096779 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0dd7bf-130c-4f38-8d09-8152348dcc84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.097387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0dd7bf-130c-4f38-8d09-8152348dcc84 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.097664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0dd7bf-130c-4f38-8d09-8152348dcc84 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.097934 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0dd7bf-130c-4f38-8d09-8152348dcc84 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.098565 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0dd7bf-130c-4f38-8d09-8152348dcc84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.100531 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0dd7bf-130c-4f38-8d09-8152348dcc84 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.137618 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1197/2367] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:28 2022] POST /identity/v3/users => generated 287 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:28.141523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207f63f5-665d-4adf-ad8c-6b3db0174c17 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.154242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207f63f5-665d-4adf-ad8c-6b3db0174c17 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.159161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae7c026-2114-4a10-8268-559ba9562033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.159779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae7c026-2114-4a10-8268-559ba9562033 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.160087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae7c026-2114-4a10-8268-559ba9562033 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.160404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae7c026-2114-4a10-8268-559ba9562033 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.161173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae7c026-2114-4a10-8268-559ba9562033 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.162863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae7c026-2114-4a10-8268-559ba9562033 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.182459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1171/2368] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:28.186567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0dd7bf-130c-4f38-8d09-8152348dcc84 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.194028 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:28.199959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0dd7bf-130c-4f38-8d09-8152348dcc84 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.208910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082c504e-5319-42dc-abff-379113a45b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.209669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082c504e-5319-42dc-abff-379113a45b43 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.209669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082c504e-5319-42dc-abff-379113a45b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.209902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082c504e-5319-42dc-abff-379113a45b43 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.210640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082c504e-5319-42dc-abff-379113a45b43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.212831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082c504e-5319-42dc-abff-379113a45b43 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.226963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1198/2369] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:28.232567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae7c026-2114-4a10-8268-559ba9562033 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.244910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae7c026-2114-4a10-8268-559ba9562033 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.251527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1e55f9-ffba-4d56-b89b-4076f61139b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.252329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1e55f9-ffba-4d56-b89b-4076f61139b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.252688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1e55f9-ffba-4d56-b89b-4076f61139b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.253034 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1e55f9-ffba-4d56-b89b-4076f61139b4 admin admin] PATH_INFO: `/v3/projects/f0331d6a185542d89e413dd4ee80c0d9/users/fc66b93c8c1d4b95a955402bef72df5d/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.270260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1e55f9-ffba-4d56-b89b-4076f61139b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0331d6a185542d89e413dd4ee80c0d9, user_id=fc66b93c8c1d4b95a955402bef72df5d, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.273072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1e55f9-ffba-4d56-b89b-4076f61139b4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.288566 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1172/2370] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:28 2022] PUT /identity/v3/projects/f0331d6a185542d89e413dd4ee80c0d9/users/fc66b93c8c1d4b95a955402bef72df5d/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:28.301424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c4e9c6-b7f7-40d0-9c13-b2f408ca1ebd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.301866 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c4e9c6-b7f7-40d0-9c13-b2f408ca1ebd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.301970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c4e9c6-b7f7-40d0-9c13-b2f408ca1ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.337026 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-52c4e9c6-b7f7-40d0-9c13-b2f408ca1ebd None None] MFA Rules not processed for user `fc66b93c8c1d4b95a955402bef72df5d`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:28.338313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-52c4e9c6-b7f7-40d0-9c13-b2f408ca1ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:28.408275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1199/2371] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:28.412861 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1e55f9-ffba-4d56-b89b-4076f61139b4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.433377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c4e9c6-b7f7-40d0-9c13-b2f408ca1ebd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.434930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-52c4e9c6-b7f7-40d0-9c13-b2f408ca1ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:28.479797 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1e55f9-ffba-4d56-b89b-4076f61139b4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.484721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6938913a-f80f-4aef-a0cf-d6ea3b508db0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.485337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6938913a-f80f-4aef-a0cf-d6ea3b508db0 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.485615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6938913a-f80f-4aef-a0cf-d6ea3b508db0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.485885 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6938913a-f80f-4aef-a0cf-d6ea3b508db0 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.486536 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6938913a-f80f-4aef-a0cf-d6ea3b508db0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.488472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6938913a-f80f-4aef-a0cf-d6ea3b508db0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.494100 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:28.507887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c4e9c6-b7f7-40d0-9c13-b2f408ca1ebd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.513725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.514360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.514667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.514946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] PATH_INFO: `/v3/users/eacd53e5a96447259ee6c4877c53e784` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.516084 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] RBAC: Authorizing `identity:delete_user(user_id=eacd53e5a96447259ee6c4877c53e784)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.518020 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.518537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1173/2372] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:28 2022] POST /identity/v3/projects => generated 373 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:28.522394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6938913a-f80f-4aef-a0cf-d6ea3b508db0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.571147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eacd53e5a96447259ee6c4877c53e784'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:28.582802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eacd53e5a96447259ee6c4877c53e784'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:28.596396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eacd53e5a96447259ee6c4877c53e784'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:28.597150 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eacd53e5a96447259ee6c4877c53e784'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:28.597311 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6938913a-f80f-4aef-a0cf-d6ea3b508db0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.597821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-594dd920-25f5-4433-a2a9-62dec3002ebf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eacd53e5a96447259ee6c4877c53e784'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:28.602608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.603179 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.603480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.603793 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.604537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.607051 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.608589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:28.613288 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1200/2373] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:28 2022] DELETE /identity/v3/users/eacd53e5a96447259ee6c4877c53e784 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:28.618067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c62387-5bb0-4550-adfe-e34376a7d83c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.618632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c62387-5bb0-4550-adfe-e34376a7d83c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.618632 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c62387-5bb0-4550-adfe-e34376a7d83c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.634144 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1174/2374] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:28 2022] POST /identity/v3/users => generated 293 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:28.635435 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.638376 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:28.641038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-69c62387-5bb0-4550-adfe-e34376a7d83c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:28.642199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-69c62387-5bb0-4550-adfe-e34376a7d83c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:28.651060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50503c-ba2f-4328-8cde-f7e4ce334e8c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.708349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1201/2375] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:28.718927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c62387-5bb0-4550-adfe-e34376a7d83c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.720434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-69c62387-5bb0-4550-adfe-e34376a7d83c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:28.732140 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0baa43d-0828-4824-bdcb-3d9fbf613760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.732769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0baa43d-0828-4824-bdcb-3d9fbf613760 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.733092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0baa43d-0828-4824-bdcb-3d9fbf613760 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.733368 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0baa43d-0828-4824-bdcb-3d9fbf613760 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.734046 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0baa43d-0828-4824-bdcb-3d9fbf613760 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.735662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0baa43d-0828-4824-bdcb-3d9fbf613760 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.747424 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1175/2376] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:28.752173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0baa43d-0828-4824-bdcb-3d9fbf613760 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.762609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0baa43d-0828-4824-bdcb-3d9fbf613760 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.767419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e41b1b-47f4-4e6a-b3b4-077517b6c76e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.768097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e41b1b-47f4-4e6a-b3b4-077517b6c76e admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.768412 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e41b1b-47f4-4e6a-b3b4-077517b6c76e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.768691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e41b1b-47f4-4e6a-b3b4-077517b6c76e admin admin] PATH_INFO: `/v3/projects/00ff9f9a976c4679882d9b38c21f6d91/users/35a45d5b6f294b2fb19acbccb570595f/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.784620 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e41b1b-47f4-4e6a-b3b4-077517b6c76e admin admin] RBAC: Authorizing `identity:create_grant(project_id=00ff9f9a976c4679882d9b38c21f6d91, user_id=35a45d5b6f294b2fb19acbccb570595f, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.787186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e41b1b-47f4-4e6a-b3b4-077517b6c76e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.788482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c62387-5bb0-4550-adfe-e34376a7d83c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.792411 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa05526-a7ab-4643-8e1c-3e545b6fd0c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.793619 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa05526-a7ab-4643-8e1c-3e545b6fd0c4 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.794040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa05526-a7ab-4643-8e1c-3e545b6fd0c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.794432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa05526-a7ab-4643-8e1c-3e545b6fd0c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.802018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1176/2377] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:28 2022] PUT /identity/v3/projects/00ff9f9a976c4679882d9b38c21f6d91/users/35a45d5b6f294b2fb19acbccb570595f/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:28.805653 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa05526-a7ab-4643-8e1c-3e545b6fd0c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.806403 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e41b1b-47f4-4e6a-b3b4-077517b6c76e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.807268 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa05526-a7ab-4643-8e1c-3e545b6fd0c4 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.818836 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1202/2378] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:28.861186 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:28.879020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e41b1b-47f4-4e6a-b3b4-077517b6c76e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.884207 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac362111-d523-4eb9-b39d-71364fb04e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.884817 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac362111-d523-4eb9-b39d-71364fb04e50 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.885129 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac362111-d523-4eb9-b39d-71364fb04e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.885356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac362111-d523-4eb9-b39d-71364fb04e50 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.886000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac362111-d523-4eb9-b39d-71364fb04e50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.888381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac362111-d523-4eb9-b39d-71364fb04e50 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.902664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1177/2379] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:28.903869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa05526-a7ab-4643-8e1c-3e545b6fd0c4 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:28.916483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa05526-a7ab-4643-8e1c-3e545b6fd0c4 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:28.921564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:28.922657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.923142 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.923556 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] PATH_INFO: `/v3/projects/00ff9f9a976c4679882d9b38c21f6d91/users/35a45d5b6f294b2fb19acbccb570595f/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:28.926320 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00ff9f9a976c4679882d9b38c21f6d91, user_id=35a45d5b6f294b2fb19acbccb570595f, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:28.929697 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:28.945133 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1203/2380] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:28 2022] PUT /identity/v3/projects/00ff9f9a976c4679882d9b38c21f6d91/users/35a45d5b6f294b2fb19acbccb570595f/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:28.962495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cee904f-9d0f-4087-bef9-d14b8c0f3cff None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:28.962782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cee904f-9d0f-4087-bef9-d14b8c0f3cff None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:28.963074 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cee904f-9d0f-4087-bef9-d14b8c0f3cff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:29.000000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7cee904f-9d0f-4087-bef9-d14b8c0f3cff None None] MFA Rules not processed for user `35a45d5b6f294b2fb19acbccb570595f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:29.000711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7cee904f-9d0f-4087-bef9-d14b8c0f3cff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:29.082429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1178/2381] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:28 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:29.094746 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:29.096549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:29.149789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cee904f-9d0f-4087-bef9-d14b8c0f3cff None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:29.151271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7cee904f-9d0f-4087-bef9-d14b8c0f3cff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:29.166584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c154a2-38b1-4eaf-b510-27e9fe1f4d97 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:29.170485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc20b89-a109-4d4b-9e57-54b516743a6a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:29.171104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc20b89-a109-4d4b-9e57-54b516743a6a service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:29.171528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc20b89-a109-4d4b-9e57-54b516743a6a service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:29.171810 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc20b89-a109-4d4b-9e57-54b516743a6a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:29.182918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc20b89-a109-4d4b-9e57-54b516743a6a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:29.184855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc20b89-a109-4d4b-9e57-54b516743a6a service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:29.196994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1204/2382] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:29 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:29.238976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cee904f-9d0f-4087-bef9-d14b8c0f3cff None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:29.247042 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:29.247743 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:29.247969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:29.248277 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] PATH_INFO: `/v3/projects/44659d3fd80449a6b525672c1e4b5195` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:29.249411 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] RBAC: Authorizing `identity:delete_project(project_id=44659d3fd80449a6b525672c1e4b5195)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:29.251305 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:29.304488 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44659d3fd80449a6b525672c1e4b5195', 'request_id': 'req-ee86d481-7cc1-4818-900a-5e57f38c972b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.312740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44659d3fd80449a6b525672c1e4b5195', 'request_id': 'req-ee86d481-7cc1-4818-900a-5e57f38c972b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.319984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44659d3fd80449a6b525672c1e4b5195', 'request_id': 'req-ee86d481-7cc1-4818-900a-5e57f38c972b'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.329744 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:29.334234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] The token cache is being invalidate because project 44659d3fd80449a6b525672c1e4b5195 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:29.334860 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.335848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.337518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1179/2383] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:29 2022] DELETE /identity/v3/projects/44659d3fd80449a6b525672c1e4b5195 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:29.342440 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc20b89-a109-4d4b-9e57-54b516743a6a service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:29.359154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc20b89-a109-4d4b-9e57-54b516743a6a service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:29.400205 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:29.434153 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:29.434796 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:29.435094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:29.435844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] PATH_INFO: `/v3/users/c9d90f58b1ab4b84a0586b7d8611503d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:29.436573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9d90f58b1ab4b84a0586b7d8611503d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:29.438547 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:29.504154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9d90f58b1ab4b84a0586b7d8611503d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.512731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9d90f58b1ab4b84a0586b7d8611503d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.534041 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9d90f58b1ab4b84a0586b7d8611503d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.536057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9d90f58b1ab4b84a0586b7d8611503d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.537151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f1c8fd25-8f3e-4d13-a56c-05bf9ad8e48f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9d90f58b1ab4b84a0586b7d8611503d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.567207 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1205/2384] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:29 2022] DELETE /identity/v3/users/c9d90f58b1ab4b84a0586b7d8611503d => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:29.676518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:29.677996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:29.691733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee86d481-7cc1-4818-900a-5e57f38c972b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:29.763531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:29.765558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:29.767124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:29.767500 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] PATH_INFO: `/v3/users/fc66b93c8c1d4b95a955402bef72df5d` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:29.769366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc66b93c8c1d4b95a955402bef72df5d)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:29.772662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:29.830449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc66b93c8c1d4b95a955402bef72df5d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.839258 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc66b93c8c1d4b95a955402bef72df5d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.857002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc66b93c8c1d4b95a955402bef72df5d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.857948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc66b93c8c1d4b95a955402bef72df5d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.858594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc66b93c8c1d4b95a955402bef72df5d'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:29.875149 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1180/2385] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:29 2022] DELETE /identity/v3/users/fc66b93c8c1d4b95a955402bef72df5d => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:29.880606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7c589-9ba1-43ca-b7ff-e8c13d9de231 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:29.880889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7c589-9ba1-43ca-b7ff-e8c13d9de231 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:29.881190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7c589-9ba1-43ca-b7ff-e8c13d9de231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:29.905374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-baa7c589-9ba1-43ca-b7ff-e8c13d9de231 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:29.909741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:29.909932 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-baa7c589-9ba1-43ca-b7ff-e8c13d9de231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:29.911638 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:29.983350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67624fd1-5e8c-4ce5-b4d8-0e32bc995fe2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:29.993294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:29.993937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:29.994216 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:29.994486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] PATH_INFO: `/v3/projects/6908414ecd914e6b83f63e14add2be34` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:29.995711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6908414ecd914e6b83f63e14add2be34)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:29.998400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:29.998559 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1206/2386] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:30.008739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa7c589-9ba1-43ca-b7ff-e8c13d9de231 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.010178 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-baa7c589-9ba1-43ca-b7ff-e8c13d9de231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:30.023894 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:30.054895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6908414ecd914e6b83f63e14add2be34', 'request_id': 'req-07353514-1995-47ec-8bb5-15c73c5fa4d9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.061121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6908414ecd914e6b83f63e14add2be34', 'request_id': 'req-07353514-1995-47ec-8bb5-15c73c5fa4d9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.068114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6908414ecd914e6b83f63e14add2be34', 'request_id': 'req-07353514-1995-47ec-8bb5-15c73c5fa4d9'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.075391 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa7c589-9ba1-43ca-b7ff-e8c13d9de231 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.080318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:30.082615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] The token cache is being invalidate because project 6908414ecd914e6b83f63e14add2be34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:30.082944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.083567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-07353514-1995-47ec-8bb5-15c73c5fa4d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.088564 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1181/2387] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:29 2022] DELETE /identity/v3/projects/6908414ecd914e6b83f63e14add2be34 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:30.091873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d0c00-e73a-4ffe-84c5-30cf46e86a0c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.092240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d0c00-e73a-4ffe-84c5-30cf46e86a0c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.092517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d0c00-e73a-4ffe-84c5-30cf46e86a0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.111986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9c7d0c00-e73a-4ffe-84c5-30cf46e86a0c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:30.113275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9c7d0c00-e73a-4ffe-84c5-30cf46e86a0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:30.130382 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.131004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.131262 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.131535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.133329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:30.178921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1182/2388] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:30.188762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909ce91a-7df1-4a65-bbc9-b2cae4be8cba None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.189315 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909ce91a-7df1-4a65-bbc9-b2cae4be8cba None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.189589 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909ce91a-7df1-4a65-bbc9-b2cae4be8cba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.202347 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.203825 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.211040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-909ce91a-7df1-4a65-bbc9-b2cae4be8cba None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:30.212322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-909ce91a-7df1-4a65-bbc9-b2cae4be8cba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:30.213528 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1207/2389] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:30.221066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1183/2390] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:30.225222 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.236238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26483d6e-c845-4b8a-8206-6f6bb60f2225 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.240989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.241814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.242095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.242422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.243057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:30.243360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.244991 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.257481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1208/2391] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 09:02:30.263299 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909ce91a-7df1-4a65-bbc9-b2cae4be8cba None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.275620 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909ce91a-7df1-4a65-bbc9-b2cae4be8cba None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.282116 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c83f680-4051-4756-af7f-99b403e81dcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.282728 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c83f680-4051-4756-af7f-99b403e81dcf admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.283013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c83f680-4051-4756-af7f-99b403e81dcf admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.283317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c83f680-4051-4756-af7f-99b403e81dcf admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.283988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c83f680-4051-4756-af7f-99b403e81dcf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.285930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c83f680-4051-4756-af7f-99b403e81dcf admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.317531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1184/2392] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:30 2022] POST /identity/v3/projects => generated 367 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:30.322067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.332833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44f15ab-1c64-4901-a4e4-e2ee041b4237 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.394948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940b67e6-1d8e-4762-b185-5efcf7c670b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.395980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b67e6-1d8e-4762-b185-5efcf7c670b5 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.396429 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b67e6-1d8e-4762-b185-5efcf7c670b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.396790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b67e6-1d8e-4762-b185-5efcf7c670b5 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.397505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940b67e6-1d8e-4762-b185-5efcf7c670b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.399810 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940b67e6-1d8e-4762-b185-5efcf7c670b5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.425645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1209/2393] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:30 2022] POST /identity/v3/users => generated 284 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:30.429913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c83f680-4051-4756-af7f-99b403e81dcf admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.440928 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c83f680-4051-4756-af7f-99b403e81dcf admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.445844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.446684 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.446806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.447036 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.447677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.449574 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.460609 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1185/2394] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:30.465172 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940b67e6-1d8e-4762-b185-5efcf7c670b5 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.475930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940b67e6-1d8e-4762-b185-5efcf7c670b5 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.482664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8809ed6-65fe-40f4-8c5a-dd78094568a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.483309 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8809ed6-65fe-40f4-8c5a-dd78094568a1 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.483600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8809ed6-65fe-40f4-8c5a-dd78094568a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.483877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8809ed6-65fe-40f4-8c5a-dd78094568a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.484708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8809ed6-65fe-40f4-8c5a-dd78094568a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.486354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8809ed6-65fe-40f4-8c5a-dd78094568a1 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.500257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1210/2395] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:30.504708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.505580 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8809ed6-65fe-40f4-8c5a-dd78094568a1 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.506771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:30.522145 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b507ca4c-38e4-4b4d-b44d-9f9ba5d0c38a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.522606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8809ed6-65fe-40f4-8c5a-dd78094568a1 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.527122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.527884 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.528385 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.528611 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] PATH_INFO: `/v3/projects/f0331d6a185542d89e413dd4ee80c0d9` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.528735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8bb110-dd2b-4289-a3f9-7b09bfe7c8c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.529494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8bb110-dd2b-4289-a3f9-7b09bfe7c8c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.529903 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8bb110-dd2b-4289-a3f9-7b09bfe7c8c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.530183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0331d6a185542d89e413dd4ee80c0d9)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.530937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8bb110-dd2b-4289-a3f9-7b09bfe7c8c2 admin admin] PATH_INFO: `/v3/projects/084db35ec38141c29b3729f69a6a9837/users/a3f23d505dc04573a73ea7a82cb0aaf7/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.532698 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.548449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8bb110-dd2b-4289-a3f9-7b09bfe7c8c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=084db35ec38141c29b3729f69a6a9837, user_id=a3f23d505dc04573a73ea7a82cb0aaf7, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.551086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8bb110-dd2b-4289-a3f9-7b09bfe7c8c2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.560917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:30.567180 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1211/2396] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:30 2022] PUT /identity/v3/projects/084db35ec38141c29b3729f69a6a9837/users/a3f23d505dc04573a73ea7a82cb0aaf7/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:30.580296 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e43ea-7889-4200-997a-5d4c81cd7fe3 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.580544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e43ea-7889-4200-997a-5d4c81cd7fe3 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.580895 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e43ea-7889-4200-997a-5d4c81cd7fe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.589089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0331d6a185542d89e413dd4ee80c0d9', 'request_id': 'req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.601804 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0331d6a185542d89e413dd4ee80c0d9', 'request_id': 'req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.611357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0331d6a185542d89e413dd4ee80c0d9', 'request_id': 'req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.615464 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:30.632385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-548e43ea-7889-4200-997a-5d4c81cd7fe3 None None] MFA Rules not processed for user `a3f23d505dc04573a73ea7a82cb0aaf7`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:30.634067 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-548e43ea-7889-4200-997a-5d4c81cd7fe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:30.635813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] The token cache is being invalidate because project f0331d6a185542d89e413dd4ee80c0d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:30.636109 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.636842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.638347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1186/2397] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:30 2022] DELETE /identity/v3/projects/f0331d6a185542d89e413dd4ee80c0d9 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:30.654678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.657220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:30.747670 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1212/2398] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:30 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:30.752459 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e43ea-7889-4200-997a-5d4c81cd7fe3 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.754446 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-548e43ea-7889-4200-997a-5d4c81cd7fe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:30.759227 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b1fdaa-b18d-4a66-8aea-3393a9dbe017 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.764664 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.765380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.765665 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.765941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] PATH_INFO: `/v3/users/35a45d5b6f294b2fb19acbccb570595f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.767284 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35a45d5b6f294b2fb19acbccb570595f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.769347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.775043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e43ea-7889-4200-997a-5d4c81cd7fe3 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:30.790147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7cad2-cf73-4e4e-821a-d5111ec47ec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:30.791546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7cad2-cf73-4e4e-821a-d5111ec47ec8 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:30.794159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7cad2-cf73-4e4e-821a-d5111ec47ec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:30.794834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7cad2-cf73-4e4e-821a-d5111ec47ec8 admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:30.795733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7cad2-cf73-4e4e-821a-d5111ec47ec8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:30.820664 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7cad2-cf73-4e4e-821a-d5111ec47ec8 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:30.843033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35a45d5b6f294b2fb19acbccb570595f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.852931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35a45d5b6f294b2fb19acbccb570595f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.860199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1213/2399] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:30 2022] POST /identity/v3/projects => generated 369 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:30.862527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7cad2-cf73-4e4e-821a-d5111ec47ec8 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:30.868710 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35a45d5b6f294b2fb19acbccb570595f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.869839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35a45d5b6f294b2fb19acbccb570595f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.872841 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35a45d5b6f294b2fb19acbccb570595f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:30.886869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1187/2400] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:30 2022] DELETE /identity/v3/users/35a45d5b6f294b2fb19acbccb570595f => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:30.937189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7cad2-cf73-4e4e-821a-d5111ec47ec8 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.008405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1dcad9-5d3d-4242-a245-299579fb3ac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.009107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1dcad9-5d3d-4242-a245-299579fb3ac2 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.009456 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1dcad9-5d3d-4242-a245-299579fb3ac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.009793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1dcad9-5d3d-4242-a245-299579fb3ac2 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.010596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1dcad9-5d3d-4242-a245-299579fb3ac2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.013740 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1dcad9-5d3d-4242-a245-299579fb3ac2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.040762 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1214/2401] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:30 2022] POST /identity/v3/users => generated 291 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:31.047281 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.048922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:31.065690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc4997f-4e80-4e13-8fd3-c67f25255c99 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.071749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ccf05e-0cbe-427e-ad5f-42df2c1fa9dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.072543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ccf05e-0cbe-427e-ad5f-42df2c1fa9dc admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.072894 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ccf05e-0cbe-427e-ad5f-42df2c1fa9dc admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.073233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ccf05e-0cbe-427e-ad5f-42df2c1fa9dc admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.074004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ccf05e-0cbe-427e-ad5f-42df2c1fa9dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.075878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ccf05e-0cbe-427e-ad5f-42df2c1fa9dc admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.087097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1188/2402] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:31 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:31.091965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1dcad9-5d3d-4242-a245-299579fb3ac2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.099189 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:31.108906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1dcad9-5d3d-4242-a245-299579fb3ac2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.113675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.114335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.114612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.115114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] PATH_INFO: `/v3/projects/a3e06fa7b4454b2fa37b92bd5412cebb/users/118d48db24584a7286bee0bf540f5560/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.132232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3e06fa7b4454b2fa37b92bd5412cebb, user_id=118d48db24584a7286bee0bf540f5560, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.134985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.149376 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1215/2403] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:31 2022] PUT /identity/v3/projects/a3e06fa7b4454b2fa37b92bd5412cebb/users/118d48db24584a7286bee0bf540f5560/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:31.154369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ccf05e-0cbe-427e-ad5f-42df2c1fa9dc admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.164601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ccf05e-0cbe-427e-ad5f-42df2c1fa9dc admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.195688 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.197604 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:31.230352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f61684-d663-424d-ae58-491c52f87798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.230777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f61684-d663-424d-ae58-491c52f87798 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.231054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f61684-d663-424d-ae58-491c52f87798 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.231329 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f61684-d663-424d-ae58-491c52f87798 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.231963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f61684-d663-424d-ae58-491c52f87798 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.233647 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f61684-d663-424d-ae58-491c52f87798 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.236240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:31.247460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1189/2404] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:31 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:31.251080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f61684-d663-424d-ae58-491c52f87798 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.261426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f61684-d663-424d-ae58-491c52f87798 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.266402 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a0350a-f993-4974-a381-e0572d103c08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.267405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a0350a-f993-4974-a381-e0572d103c08 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.267747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a0350a-f993-4974-a381-e0572d103c08 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.268683 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a0350a-f993-4974-a381-e0572d103c08 admin admin] PATH_INFO: `/v3/projects/a3e06fa7b4454b2fa37b92bd5412cebb/users/118d48db24584a7286bee0bf540f5560/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.270646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a0350a-f993-4974-a381-e0572d103c08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3e06fa7b4454b2fa37b92bd5412cebb, user_id=118d48db24584a7286bee0bf540f5560, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.271751 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f62da4-8ce7-4304-840b-93e65ff3265e admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.273345 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a0350a-f993-4974-a381-e0572d103c08 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.276521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.277875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.278158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.278417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] PATH_INFO: `/v3/projects/00ff9f9a976c4679882d9b38c21f6d91` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.279538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=00ff9f9a976c4679882d9b38c21f6d91)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.281497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.285852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1190/2405] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:31 2022] PUT /identity/v3/projects/a3e06fa7b4454b2fa37b92bd5412cebb/users/118d48db24584a7286bee0bf540f5560/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:31.296783 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4682d17-d3db-45ed-a73b-8babbe40aa86 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.297065 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4682d17-d3db-45ed-a73b-8babbe40aa86 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.297338 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4682d17-d3db-45ed-a73b-8babbe40aa86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.329054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e4682d17-d3db-45ed-a73b-8babbe40aa86 None None] MFA Rules not processed for user `118d48db24584a7286bee0bf540f5560`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:31.330267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e4682d17-d3db-45ed-a73b-8babbe40aa86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:31.336906 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00ff9f9a976c4679882d9b38c21f6d91', 'request_id': 'req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:31.343788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00ff9f9a976c4679882d9b38c21f6d91', 'request_id': 'req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:31.351838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00ff9f9a976c4679882d9b38c21f6d91', 'request_id': 'req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:31.363783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] The token cache is being invalidate because project 00ff9f9a976c4679882d9b38c21f6d91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:31.364076 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:31.364764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-1a3d24c3-4af6-40ed-a6e3-2bb86c9c1e8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:31.366821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1216/2406] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:31 2022] DELETE /identity/v3/projects/00ff9f9a976c4679882d9b38c21f6d91 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:31.371863 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d62dbe-2d4a-4211-98de-4148e1a0c5d8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.372241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d62dbe-2d4a-4211-98de-4148e1a0c5d8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.372448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d62dbe-2d4a-4211-98de-4148e1a0c5d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.391829 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-87d62dbe-2d4a-4211-98de-4148e1a0c5d8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:31.393069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-87d62dbe-2d4a-4211-98de-4148e1a0c5d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:31.403354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1191/2407] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:31 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:31.414546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4682d17-d3db-45ed-a73b-8babbe40aa86 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.415886 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e4682d17-d3db-45ed-a73b-8babbe40aa86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:31.454151 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1217/2408] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:31.462079 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ac592-d106-4696-bf66-498246de0b07 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.462365 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ac592-d106-4696-bf66-498246de0b07 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.462630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ac592-d106-4696-bf66-498246de0b07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.474080 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4682d17-d3db-45ed-a73b-8babbe40aa86 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.477752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e52cf26-011f-403c-9cc1-4a919603050f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.478356 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52cf26-011f-403c-9cc1-4a919603050f service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.478636 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52cf26-011f-403c-9cc1-4a919603050f service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.478910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52cf26-011f-403c-9cc1-4a919603050f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.483416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-9f3ac592-d106-4696-bf66-498246de0b07 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:31.484720 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9f3ac592-d106-4696-bf66-498246de0b07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:31.488969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52cf26-011f-403c-9cc1-4a919603050f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.490405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52cf26-011f-403c-9cc1-4a919603050f service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.494009 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1218/2409] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:31.498358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3ac592-d106-4696-bf66-498246de0b07 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.500960 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1192/2410] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:31 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:31.504609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:31.508390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3ac592-d106-4696-bf66-498246de0b07 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.515695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.516451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.516909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.517385 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.518247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:31.518678 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.520317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.533097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1219/2411] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:31.537603 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e52cf26-011f-403c-9cc1-4a919603050f service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.551680 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e52cf26-011f-403c-9cc1-4a919603050f service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.556420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c920e1e4-06fe-4d8b-8d1e-53edba4fd436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.557018 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c920e1e4-06fe-4d8b-8d1e-53edba4fd436 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.557296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c920e1e4-06fe-4d8b-8d1e-53edba4fd436 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.557569 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c920e1e4-06fe-4d8b-8d1e-53edba4fd436 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c920e1e4-06fe-4d8b-8d1e-53edba4fd436 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.560126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c920e1e4-06fe-4d8b-8d1e-53edba4fd436 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.595189 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1193/2412] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:31 2022] POST /identity/v3/projects => generated 387 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:31.596294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.606856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b01a296-756b-43c3-b8ed-10f077c5bc10 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.669642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586183a-07a0-48b2-aaf1-01da3a077398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.670256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586183a-07a0-48b2-aaf1-01da3a077398 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.670598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586183a-07a0-48b2-aaf1-01da3a077398 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.670872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586183a-07a0-48b2-aaf1-01da3a077398 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.671543 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5586183a-07a0-48b2-aaf1-01da3a077398 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.673694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5586183a-07a0-48b2-aaf1-01da3a077398 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.695708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1220/2413] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:31 2022] POST /identity/v3/users => generated 294 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:31.700287 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c920e1e4-06fe-4d8b-8d1e-53edba4fd436 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.712628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c920e1e4-06fe-4d8b-8d1e-53edba4fd436 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.717929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9006ef7-265f-451b-99d1-1f2f4840bccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.718525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9006ef7-265f-451b-99d1-1f2f4840bccc admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.718802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9006ef7-265f-451b-99d1-1f2f4840bccc admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.719069 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9006ef7-265f-451b-99d1-1f2f4840bccc admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.719770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9006ef7-265f-451b-99d1-1f2f4840bccc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.721381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9006ef7-265f-451b-99d1-1f2f4840bccc admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.732991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1194/2414] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:31.736985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586183a-07a0-48b2-aaf1-01da3a077398 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.747247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586183a-07a0-48b2-aaf1-01da3a077398 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.751882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a632406-b2a5-42a5-9640-da46ae58c801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.752518 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a632406-b2a5-42a5-9640-da46ae58c801 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.752798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a632406-b2a5-42a5-9640-da46ae58c801 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.753073 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a632406-b2a5-42a5-9640-da46ae58c801 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.754046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a632406-b2a5-42a5-9640-da46ae58c801 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.756269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a632406-b2a5-42a5-9640-da46ae58c801 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.768161 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1221/2415] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:31.772540 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9006ef7-265f-451b-99d1-1f2f4840bccc admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:31.783773 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9006ef7-265f-451b-99d1-1f2f4840bccc admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:31.788614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5100785-94f8-4f0d-afb8-3bff8697c665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:31.789265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5100785-94f8-4f0d-afb8-3bff8697c665 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.789549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5100785-94f8-4f0d-afb8-3bff8697c665 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.789827 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5100785-94f8-4f0d-afb8-3bff8697c665 admin admin] PATH_INFO: `/v3/projects/d0f89c2490d84553a72454874848f9eb/users/f1d9f160d3564acc816c92c5d1549c72/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.811145 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5100785-94f8-4f0d-afb8-3bff8697c665 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0f89c2490d84553a72454874848f9eb, user_id=f1d9f160d3564acc816c92c5d1549c72, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:31.814063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5100785-94f8-4f0d-afb8-3bff8697c665 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:31.831752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1195/2416] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:31 2022] PUT /identity/v3/projects/d0f89c2490d84553a72454874848f9eb/users/f1d9f160d3564acc816c92c5d1549c72/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:31.850104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf1410-b6d3-4641-abe5-b73ae1c0b85c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:31.850104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf1410-b6d3-4641-abe5-b73ae1c0b85c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:31.850104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf1410-b6d3-4641-abe5-b73ae1c0b85c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:31.897166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e8bf1410-b6d3-4641-abe5-b73ae1c0b85c None None] MFA Rules not processed for user `f1d9f160d3564acc816c92c5d1549c72`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:31.898658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e8bf1410-b6d3-4641-abe5-b73ae1c0b85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:31.995418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1222/2417] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:31 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:31.997634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5100785-94f8-4f0d-afb8-3bff8697c665 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:32.026372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:32.084330 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5100785-94f8-4f0d-afb8-3bff8697c665 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:32.091563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b6d1b8-11e0-4860-97e8-083cd84dd5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:32.091563 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b6d1b8-11e0-4860-97e8-083cd84dd5d4 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.091879 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b6d1b8-11e0-4860-97e8-083cd84dd5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.091978 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b6d1b8-11e0-4860-97e8-083cd84dd5d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.092652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b6d1b8-11e0-4860-97e8-083cd84dd5d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:32.094568 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b6d1b8-11e0-4860-97e8-083cd84dd5d4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:32.125667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1196/2418] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:31 2022] POST /identity/v3/projects => generated 387 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:32.130068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf1410-b6d3-4641-abe5-b73ae1c0b85c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:32.142479 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf1410-b6d3-4641-abe5-b73ae1c0b85c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:32.216302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498fe73d-f940-4255-9f27-5d593c486e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:32.217245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498fe73d-f940-4255-9f27-5d593c486e87 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.217507 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498fe73d-f940-4255-9f27-5d593c486e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.217851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498fe73d-f940-4255-9f27-5d593c486e87 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.218640 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498fe73d-f940-4255-9f27-5d593c486e87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:32.220806 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498fe73d-f940-4255-9f27-5d593c486e87 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:32.229861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:32.254054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b6d1b8-11e0-4860-97e8-083cd84dd5d4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:32.254248 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1223/2419] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:32 2022] POST /identity/v3/users => generated 300 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:32.268637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b6d1b8-11e0-4860-97e8-083cd84dd5d4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:32.274359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15be4f24-f17b-4ee3-b4d6-43c9cd2a71e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:32.274996 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15be4f24-f17b-4ee3-b4d6-43c9cd2a71e6 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.275275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15be4f24-f17b-4ee3-b4d6-43c9cd2a71e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.275549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15be4f24-f17b-4ee3-b4d6-43c9cd2a71e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.276275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15be4f24-f17b-4ee3-b4d6-43c9cd2a71e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:32.278491 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15be4f24-f17b-4ee3-b4d6-43c9cd2a71e6 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:32.295862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1197/2420] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:32 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:32.297121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498fe73d-f940-4255-9f27-5d593c486e87 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:32.309736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498fe73d-f940-4255-9f27-5d593c486e87 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:32.315088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46599a-694f-4fe0-a421-5a1e375001c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:32.316016 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46599a-694f-4fe0-a421-5a1e375001c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.316541 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46599a-694f-4fe0-a421-5a1e375001c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.317022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46599a-694f-4fe0-a421-5a1e375001c4 admin admin] PATH_INFO: `/v3/projects/8f8a8b47b5de40daa0d478f42d1db8aa/users/1913dd0ca6ad46fcb2eb65470640961d/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.341668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f46599a-694f-4fe0-a421-5a1e375001c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f8a8b47b5de40daa0d478f42d1db8aa, user_id=1913dd0ca6ad46fcb2eb65470640961d, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:32.344439 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f46599a-694f-4fe0-a421-5a1e375001c4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:32.361406 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1224/2421] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:32 2022] PUT /identity/v3/projects/8f8a8b47b5de40daa0d478f42d1db8aa/users/1913dd0ca6ad46fcb2eb65470640961d/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:32.365493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15be4f24-f17b-4ee3-b4d6-43c9cd2a71e6 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:32.393980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15be4f24-f17b-4ee3-b4d6-43c9cd2a71e6 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:32.468596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25b0c5a-4030-4603-be9b-07c4b26582c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:32.472307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b0c5a-4030-4603-be9b-07c4b26582c7 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.472812 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b0c5a-4030-4603-be9b-07c4b26582c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.473106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b0c5a-4030-4603-be9b-07c4b26582c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.474866 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25b0c5a-4030-4603-be9b-07c4b26582c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:32.475617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25b0c5a-4030-4603-be9b-07c4b26582c7 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:32.491071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1198/2422] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:32 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:32.492622 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46599a-694f-4fe0-a421-5a1e375001c4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:32.507219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46599a-694f-4fe0-a421-5a1e375001c4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:32.514312 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304d534-5933-4c4c-9522-1537cdea9fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:32.515263 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304d534-5933-4c4c-9522-1537cdea9fc2 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.515642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304d534-5933-4c4c-9522-1537cdea9fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.516247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304d534-5933-4c4c-9522-1537cdea9fc2 admin admin] PATH_INFO: `/v3/projects/8f8a8b47b5de40daa0d478f42d1db8aa/users/1913dd0ca6ad46fcb2eb65470640961d/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.522285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304d534-5933-4c4c-9522-1537cdea9fc2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f8a8b47b5de40daa0d478f42d1db8aa, user_id=1913dd0ca6ad46fcb2eb65470640961d, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:32.522285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304d534-5933-4c4c-9522-1537cdea9fc2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:32.534008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1225/2423] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:32 2022] PUT /identity/v3/projects/8f8a8b47b5de40daa0d478f42d1db8aa/users/1913dd0ca6ad46fcb2eb65470640961d/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:32.546296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4372cdc-0f90-419d-8525-a1c5dfa521b8 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.546615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4372cdc-0f90-419d-8525-a1c5dfa521b8 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.546902 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4372cdc-0f90-419d-8525-a1c5dfa521b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.579599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-f4372cdc-0f90-419d-8525-a1c5dfa521b8 None None] MFA Rules not processed for user `1913dd0ca6ad46fcb2eb65470640961d`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:32.580888 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f4372cdc-0f90-419d-8525-a1c5dfa521b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:32.666333 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1199/2424] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:32 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:32.690092 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304d534-5933-4c4c-9522-1537cdea9fc2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:32.748822 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304d534-5933-4c4c-9522-1537cdea9fc2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:32.752804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8598c0-7baa-4ae5-8517-41e583398857 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:32.753397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8598c0-7baa-4ae5-8517-41e583398857 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.753672 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8598c0-7baa-4ae5-8517-41e583398857 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.754190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8598c0-7baa-4ae5-8517-41e583398857 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.763945 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8598c0-7baa-4ae5-8517-41e583398857 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:32.765402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8598c0-7baa-4ae5-8517-41e583398857 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:32.774933 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1226/2425] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:32 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:32.805486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4372cdc-0f90-419d-8525-a1c5dfa521b8 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:32.819922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:32.884197 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4372cdc-0f90-419d-8525-a1c5dfa521b8 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:32.890796 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:32.891573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:32.892115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:32.892819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] PATH_INFO: `/v3/users/f1d9f160d3564acc816c92c5d1549c72` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:32.894760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1d9f160d3564acc816c92c5d1549c72)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:32.897002 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:32.962306 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1d9f160d3564acc816c92c5d1549c72'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:32.970673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1d9f160d3564acc816c92c5d1549c72'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.001428 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1d9f160d3564acc816c92c5d1549c72'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.002176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1d9f160d3564acc816c92c5d1549c72'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.002800 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1d9f160d3564acc816c92c5d1549c72'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.023010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1200/2426] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:32 2022] DELETE /identity/v3/users/f1d9f160d3564acc816c92c5d1549c72 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:33.027361 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa931a9d-43a4-434a-b84e-9e4845df3788 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:33.027691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa931a9d-43a4-434a-b84e-9e4845df3788 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:33.027990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa931a9d-43a4-434a-b84e-9e4845df3788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:33.049633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fa931a9d-43a4-434a-b84e-9e4845df3788 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:33.050902 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fa931a9d-43a4-434a-b84e-9e4845df3788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:33.118500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1227/2427] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:33.129219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:33.130747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:33.194302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93515ce7-6e93-4934-a977-fe288fcf5308 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:33.199634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51dfdc9-6f77-4bdb-accd-1d784549a5af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:33.199634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51dfdc9-6f77-4bdb-accd-1d784549a5af service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:33.199863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51dfdc9-6f77-4bdb-accd-1d784549a5af service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:33.199945 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51dfdc9-6f77-4bdb-accd-1d784549a5af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:33.210327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51dfdc9-6f77-4bdb-accd-1d784549a5af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:33.212362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51dfdc9-6f77-4bdb-accd-1d784549a5af service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:33.222972 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1201/2428] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:33.263788 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa931a9d-43a4-434a-b84e-9e4845df3788 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:33.265371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fa931a9d-43a4-434a-b84e-9e4845df3788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:33.279450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa931a9d-43a4-434a-b84e-9e4845df3788 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:33.285228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:33.285891 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:33.286175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:33.286449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] PATH_INFO: `/v3/users/a3f23d505dc04573a73ea7a82cb0aaf7` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:33.287581 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3f23d505dc04573a73ea7a82cb0aaf7)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:33.289864 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:33.346629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3f23d505dc04573a73ea7a82cb0aaf7'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.348677 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:INSERT=5,keystone:SELECT=88,keystone:DELETE=7 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:33.354798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3f23d505dc04573a73ea7a82cb0aaf7'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.369071 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3f23d505dc04573a73ea7a82cb0aaf7'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.369935 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3f23d505dc04573a73ea7a82cb0aaf7'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.371157 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3f23d505dc04573a73ea7a82cb0aaf7'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.386175 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1228/2429] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:33 2022] DELETE /identity/v3/users/a3f23d505dc04573a73ea7a82cb0aaf7 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:33.391230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c892d4c-b079-484a-8b0e-b942fed4cd54 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:33.391572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c892d4c-b079-484a-8b0e-b942fed4cd54 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:33.391855 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c892d4c-b079-484a-8b0e-b942fed4cd54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:33.413948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-9c892d4c-b079-484a-8b0e-b942fed4cd54 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:33.415370 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9c892d4c-b079-484a-8b0e-b942fed4cd54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:33.449429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:33.482173 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1202/2430] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:33.492337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:33.493778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:33.540413 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c892d4c-b079-484a-8b0e-b942fed4cd54 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:33.542377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-9c892d4c-b079-484a-8b0e-b942fed4cd54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:33.555966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c892d4c-b079-484a-8b0e-b942fed4cd54 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:33.561701 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:33.562481 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:33.562828 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:33.563236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] PATH_INFO: `/v3/projects/d0f89c2490d84553a72454874848f9eb` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:33.564429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0f89c2490d84553a72454874848f9eb)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:33.566461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:33.569808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323ff14f-5eac-4740-bcb4-bc04df0f8f5e admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:33.574881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3f774e-061f-4131-b0bb-15f003c22e6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:33.575744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f774e-061f-4131-b0bb-15f003c22e6a service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:33.576025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f774e-061f-4131-b0bb-15f003c22e6a service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:33.576320 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f774e-061f-4131-b0bb-15f003c22e6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:33.586297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3f774e-061f-4131-b0bb-15f003c22e6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:33.587739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3f774e-061f-4131-b0bb-15f003c22e6a service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:33.599267 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1229/2431] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:33.619920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0f89c2490d84553a72454874848f9eb', 'request_id': 'req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.626948 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0f89c2490d84553a72454874848f9eb', 'request_id': 'req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.633301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0f89c2490d84553a72454874848f9eb', 'request_id': 'req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.646534 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] The token cache is being invalidate because project d0f89c2490d84553a72454874848f9eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:33.646721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.647367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.648920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1203/2432] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:33 2022] DELETE /identity/v3/projects/d0f89c2490d84553a72454874848f9eb => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:33.653476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3f774e-061f-4131-b0bb-15f003c22e6a service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:33.664727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3f774e-061f-4131-b0bb-15f003c22e6a service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:33.728094 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:33.728780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:33.729064 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:33.729341 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] PATH_INFO: `/v3/users/1913dd0ca6ad46fcb2eb65470640961d` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:33.730787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1913dd0ca6ad46fcb2eb65470640961d)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:33.732769 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:33.785671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1913dd0ca6ad46fcb2eb65470640961d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.793798 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1913dd0ca6ad46fcb2eb65470640961d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.804947 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:33.806292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1913dd0ca6ad46fcb2eb65470640961d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.807466 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1913dd0ca6ad46fcb2eb65470640961d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.810233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1913dd0ca6ad46fcb2eb65470640961d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:33.825708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1230/2433] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:33 2022] DELETE /identity/v3/users/1913dd0ca6ad46fcb2eb65470640961d => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:33.892673 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:33.895519 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:33.911798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aff8d19-b8ae-4596-ab4c-38f62dbbe17f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:33.984399 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:33.985264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:33.985633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:33.985997 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] PATH_INFO: `/v3/projects/084db35ec38141c29b3729f69a6a9837` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:33.993702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=084db35ec38141c29b3729f69a6a9837)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:33.995936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.060858 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '084db35ec38141c29b3729f69a6a9837', 'request_id': 'req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.069696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '084db35ec38141c29b3729f69a6a9837', 'request_id': 'req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.070023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:34.080362 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '084db35ec38141c29b3729f69a6a9837', 'request_id': 'req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.109298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] The token cache is being invalidate because project 084db35ec38141c29b3729f69a6a9837 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:34.109709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.110405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.111934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1204/2434] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:33 2022] DELETE /identity/v3/projects/084db35ec38141c29b3729f69a6a9837 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:34.117454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.119163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.134607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef36384-3c6c-42bb-8809-2f245d8e83b8 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.159670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.161593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.211131 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.211915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.212333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.212673 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] PATH_INFO: `/v3/users/118d48db24584a7286bee0bf540f5560` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.213901 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] RBAC: Authorizing `identity:delete_user(user_id=118d48db24584a7286bee0bf540f5560)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.215961 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.240165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6cc899-1b3c-4dc7-a364-37fdd074fc70 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.246254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.246959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.247360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.247605 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] PATH_INFO: `/v3/projects/8f8a8b47b5de40daa0d478f42d1db8aa` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.248921 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f8a8b47b5de40daa0d478f42d1db8aa)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.251161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.270571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '118d48db24584a7286bee0bf540f5560'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.278499 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '118d48db24584a7286bee0bf540f5560'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.290844 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '118d48db24584a7286bee0bf540f5560'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.291566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '118d48db24584a7286bee0bf540f5560'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.292198 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ea12bb06-7fae-4e64-b26c-0bf2a9fcf63a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '118d48db24584a7286bee0bf540f5560'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.300903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f8a8b47b5de40daa0d478f42d1db8aa', 'request_id': 'req-87927c69-c771-405b-b5f9-85f8b7e74ece'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.306250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1231/2435] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:34 2022] DELETE /identity/v3/users/118d48db24584a7286bee0bf540f5560 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:34.306799 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f8a8b47b5de40daa0d478f42d1db8aa', 'request_id': 'req-87927c69-c771-405b-b5f9-85f8b7e74ece'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.316503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f8a8b47b5de40daa0d478f42d1db8aa', 'request_id': 'req-87927c69-c771-405b-b5f9-85f8b7e74ece'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.331209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] The token cache is being invalidate because project 8f8a8b47b5de40daa0d478f42d1db8aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:34.331504 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.332327 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-87927c69-c771-405b-b5f9-85f8b7e74ece admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.333839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1205/2436] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:34 2022] DELETE /identity/v3/projects/8f8a8b47b5de40daa0d478f42d1db8aa => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:34.338831 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccfcdb9-4e41-4536-8612-315d17762f68 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.339113 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccfcdb9-4e41-4536-8612-315d17762f68 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.339415 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccfcdb9-4e41-4536-8612-315d17762f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.360733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-eccfcdb9-4e41-4536-8612-315d17762f68 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:34.361950 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-eccfcdb9-4e41-4536-8612-315d17762f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.424718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1232/2437] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:34.432312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4271e2-1d80-42fa-989f-8adcb5d92695 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.432594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4271e2-1d80-42fa-989f-8adcb5d92695 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.432862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4271e2-1d80-42fa-989f-8adcb5d92695 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.457048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-4e4271e2-1d80-42fa-989f-8adcb5d92695 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:34.458437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4e4271e2-1d80-42fa-989f-8adcb5d92695 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.467965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1206/2438] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:34.473941 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccfcdb9-4e41-4536-8612-315d17762f68 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.484651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccfcdb9-4e41-4536-8612-315d17762f68 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.489424 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.490029 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.490318 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.490592 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.491287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:34.491845 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.493492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.496735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:34.508616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1233/2439] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:34.512973 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4271e2-1d80-42fa-989f-8adcb5d92695 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.522211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4271e2-1d80-42fa-989f-8adcb5d92695 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.526895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.527517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.527798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.528072 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.528749 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.530646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.560174 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1207/2440] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:34 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:34.564255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.575154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74105642-d5e9-40e6-80ae-3f0a707e65f5 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.597001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.598925 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.650330 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f132364-c340-400e-8c42-690787633ed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.651085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f132364-c340-400e-8c42-690787633ed9 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.651465 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f132364-c340-400e-8c42-690787633ed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.651915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f132364-c340-400e-8c42-690787633ed9 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.652092 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:34.652979 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f132364-c340-400e-8c42-690787633ed9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.655648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f132364-c340-400e-8c42-690787633ed9 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.669624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e58544-ba7b-4c72-8b12-cff8fd4ab11a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.674774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.675407 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.675687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.675962 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] PATH_INFO: `/v3/projects/a3e06fa7b4454b2fa37b92bd5412cebb` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.677117 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3e06fa7b4454b2fa37b92bd5412cebb)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.679012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.679140 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1234/2441] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:34 2022] POST /identity/v3/users => generated 293 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:34.683045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f132364-c340-400e-8c42-690787633ed9 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.693048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f132364-c340-400e-8c42-690787633ed9 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.699631 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdddd8a-0235-4532-b78f-31225612657d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.700266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdddd8a-0235-4532-b78f-31225612657d admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.700573 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdddd8a-0235-4532-b78f-31225612657d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.700897 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdddd8a-0235-4532-b78f-31225612657d admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.701948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdddd8a-0235-4532-b78f-31225612657d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.703134 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdddd8a-0235-4532-b78f-31225612657d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.715494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1235/2442] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:34.719540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdddd8a-0235-4532-b78f-31225612657d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.729706 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdddd8a-0235-4532-b78f-31225612657d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.731269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3e06fa7b4454b2fa37b92bd5412cebb', 'request_id': 'req-f173b0bb-83af-4220-b9a6-f9c085123861'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.737759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3e06fa7b4454b2fa37b92bd5412cebb', 'request_id': 'req-f173b0bb-83af-4220-b9a6-f9c085123861'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.744634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3e06fa7b4454b2fa37b92bd5412cebb', 'request_id': 'req-f173b0bb-83af-4220-b9a6-f9c085123861'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.759926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] The token cache is being invalidate because project a3e06fa7b4454b2fa37b92bd5412cebb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:34.760250 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.760917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f173b0bb-83af-4220-b9a6-f9c085123861 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:34.762503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1208/2443] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:34 2022] DELETE /identity/v3/projects/a3e06fa7b4454b2fa37b92bd5412cebb => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:34.767321 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b531c-c080-41ab-a3a1-078184d9c697 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.767592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b531c-c080-41ab-a3a1-078184d9c697 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.767862 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b531c-c080-41ab-a3a1-078184d9c697 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.788693 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7b4b531c-c080-41ab-a3a1-078184d9c697 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:34.789922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7b4b531c-c080-41ab-a3a1-078184d9c697 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.795200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.795810 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.796088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.796390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.797025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.798602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.809529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1236/2444] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:34 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:34.813564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.815350 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.828660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82332af-1494-4ef8-b46c-c41ca313d76c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.833454 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e79f1a-e514-4871-a059-bee06f86751a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.834104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e79f1a-e514-4871-a059-bee06f86751a admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.834389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e79f1a-e514-4871-a059-bee06f86751a admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.834671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e79f1a-e514-4871-a059-bee06f86751a admin admin] PATH_INFO: `/v3/projects/eb933a8adba1470ab329e6e6fae57d0b/users/a6002a5b54b64f5b9f15eac86199387a/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.850281 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e79f1a-e514-4871-a059-bee06f86751a admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb933a8adba1470ab329e6e6fae57d0b, user_id=a6002a5b54b64f5b9f15eac86199387a, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.853398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e79f1a-e514-4871-a059-bee06f86751a admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.854004 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1209/2445] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:34.863285 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f13348b-ecca-4c2f-8cf6-87e403cf1de0 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.863655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f13348b-ecca-4c2f-8cf6-87e403cf1de0 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.864013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f13348b-ecca-4c2f-8cf6-87e403cf1de0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.868154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1237/2446] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:34 2022] PUT /identity/v3/projects/eb933a8adba1470ab329e6e6fae57d0b/users/a6002a5b54b64f5b9f15eac86199387a/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:34.880059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee93240-9f6d-4dbf-bbba-34f67bff2153 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.880522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee93240-9f6d-4dbf-bbba-34f67bff2153 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.880804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee93240-9f6d-4dbf-bbba-34f67bff2153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.885078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-8f13348b-ecca-4c2f-8cf6-87e403cf1de0 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:34.887252 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-8f13348b-ecca-4c2f-8cf6-87e403cf1de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.909735 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-fee93240-9f6d-4dbf-bbba-34f67bff2153 None None] MFA Rules not processed for user `a6002a5b54b64f5b9f15eac86199387a`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:34.910965 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fee93240-9f6d-4dbf-bbba-34f67bff2153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:34.945112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:34.951081 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1210/2447] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:34.955459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f13348b-ecca-4c2f-8cf6-87e403cf1de0 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:34.965498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f13348b-ecca-4c2f-8cf6-87e403cf1de0 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:34.980077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:34.981472 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:34.981747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:34.982023 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:34.982857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:34.983101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:34.984748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:34.986679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1238/2448] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:34 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:34.998204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1211/2449] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 09:02:35.001663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee93240-9f6d-4dbf-bbba-34f67bff2153 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.002394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.003232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-fee93240-9f6d-4dbf-bbba-34f67bff2153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:35.012434 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f36f7d-c186-45c0-9660-4e35f37649bf admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.017255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69df1864-1dbf-4352-97d9-908d19d8359c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.017757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69df1864-1dbf-4352-97d9-908d19d8359c admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.018094 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69df1864-1dbf-4352-97d9-908d19d8359c admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.018367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69df1864-1dbf-4352-97d9-908d19d8359c admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.019021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69df1864-1dbf-4352-97d9-908d19d8359c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.020959 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69df1864-1dbf-4352-97d9-908d19d8359c admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.025863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:35.052336 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1212/2450] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:35 2022] POST /identity/v3/projects => generated 377 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:35.056347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69df1864-1dbf-4352-97d9-908d19d8359c admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.067409 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee93240-9f6d-4dbf-bbba-34f67bff2153 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.121888 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025466ba-b6b3-4473-b592-64d1b59872b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.122641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025466ba-b6b3-4473-b592-64d1b59872b4 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.122923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025466ba-b6b3-4473-b592-64d1b59872b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.123334 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025466ba-b6b3-4473-b592-64d1b59872b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.131209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69df1864-1dbf-4352-97d9-908d19d8359c admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.137898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4f330e-ac03-40d0-8a60-de862dbf80d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.138630 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4f330e-ac03-40d0-8a60-de862dbf80d9 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.138910 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4f330e-ac03-40d0-8a60-de862dbf80d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.139372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4f330e-ac03-40d0-8a60-de862dbf80d9 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.139805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4f330e-ac03-40d0-8a60-de862dbf80d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.141745 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4f330e-ac03-40d0-8a60-de862dbf80d9 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.176868 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1213/2451] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:35 2022] POST /identity/v3/users => generated 289 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:35.181108 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4f330e-ac03-40d0-8a60-de862dbf80d9 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.189682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025466ba-b6b3-4473-b592-64d1b59872b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.191125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025466ba-b6b3-4473-b592-64d1b59872b4 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.191571 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4f330e-ac03-40d0-8a60-de862dbf80d9 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.196285 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1cb113-8ab5-4929-9107-e3550bd320b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.196830 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cb113-8ab5-4929-9107-e3550bd320b5 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.197107 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cb113-8ab5-4929-9107-e3550bd320b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.197379 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cb113-8ab5-4929-9107-e3550bd320b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.198015 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1cb113-8ab5-4929-9107-e3550bd320b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.200248 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1cb113-8ab5-4929-9107-e3550bd320b5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.201130 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1239/2452] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:35 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:35.212200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1214/2453] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:35.215115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025466ba-b6b3-4473-b592-64d1b59872b4 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.220567 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1cb113-8ab5-4929-9107-e3550bd320b5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.225918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025466ba-b6b3-4473-b592-64d1b59872b4 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.231066 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1cb113-8ab5-4929-9107-e3550bd320b5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.231306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1addc54-20c4-4b14-85d4-659a6beff5e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.231910 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1addc54-20c4-4b14-85d4-659a6beff5e4 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.232219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1addc54-20c4-4b14-85d4-659a6beff5e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.232495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1addc54-20c4-4b14-85d4-659a6beff5e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.233129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1addc54-20c4-4b14-85d4-659a6beff5e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.234704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1addc54-20c4-4b14-85d4-659a6beff5e4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.235852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.236497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.236782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.237059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] PATH_INFO: `/v3/users/a6002a5b54b64f5b9f15eac86199387a` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.238200 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6002a5b54b64f5b9f15eac86199387a)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.240099 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.246096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1240/2454] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:35.250435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1addc54-20c4-4b14-85d4-659a6beff5e4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.260539 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1addc54-20c4-4b14-85d4-659a6beff5e4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.266057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3e1c2-da29-464b-ad8b-5af1ded8e00d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.266705 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3e1c2-da29-464b-ad8b-5af1ded8e00d admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.266984 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3e1c2-da29-464b-ad8b-5af1ded8e00d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.267255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3e1c2-da29-464b-ad8b-5af1ded8e00d admin admin] PATH_INFO: `/v3/projects/ff7c2d77b80e4571b5dbec9402bbe154/users/461a9ee13a924592ab16c58f36c0aff5/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.284764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a3e1c2-da29-464b-ad8b-5af1ded8e00d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff7c2d77b80e4571b5dbec9402bbe154, user_id=461a9ee13a924592ab16c58f36c0aff5, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.287714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a3e1c2-da29-464b-ad8b-5af1ded8e00d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.296008 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6002a5b54b64f5b9f15eac86199387a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.301717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1241/2455] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:35 2022] PUT /identity/v3/projects/ff7c2d77b80e4571b5dbec9402bbe154/users/461a9ee13a924592ab16c58f36c0aff5/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:35.308064 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6002a5b54b64f5b9f15eac86199387a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.312278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15087d61-e714-46d8-af2f-bc5ac33b16b0 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.312591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15087d61-e714-46d8-af2f-bc5ac33b16b0 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.312873 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15087d61-e714-46d8-af2f-bc5ac33b16b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.321270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6002a5b54b64f5b9f15eac86199387a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.322000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6002a5b54b64f5b9f15eac86199387a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.324662 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-b959008f-96c5-4c24-9c38-bcfaa7489965 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6002a5b54b64f5b9f15eac86199387a'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.335861 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:35.345378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1215/2456] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:35 2022] DELETE /identity/v3/users/a6002a5b54b64f5b9f15eac86199387a => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:35.347938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-15087d61-e714-46d8-af2f-bc5ac33b16b0 None None] MFA Rules not processed for user `461a9ee13a924592ab16c58f36c0aff5`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:35.349549 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-15087d61-e714-46d8-af2f-bc5ac33b16b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:35.349696 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69d208f-c3cf-4a4c-83ec-fa7b2bfcab3d None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.349969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69d208f-c3cf-4a4c-83ec-fa7b2bfcab3d None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.350240 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69d208f-c3cf-4a4c-83ec-fa7b2bfcab3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.369882 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e69d208f-c3cf-4a4c-83ec-fa7b2bfcab3d None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:35.371906 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e69d208f-c3cf-4a4c-83ec-fa7b2bfcab3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:35.403593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:35.416475 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1242/2457] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:35 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:35.421228 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15087d61-e714-46d8-af2f-bc5ac33b16b0 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.422692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-15087d61-e714-46d8-af2f-bc5ac33b16b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:35.434543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1216/2458] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:35.444293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69d208f-c3cf-4a4c-83ec-fa7b2bfcab3d None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.445771 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e69d208f-c3cf-4a4c-83ec-fa7b2bfcab3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:35.496021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15087d61-e714-46d8-af2f-bc5ac33b16b0 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.500948 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25abe2c-ba39-4cba-83f7-3950f482bbab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.501572 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25abe2c-ba39-4cba-83f7-3950f482bbab admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.501869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25abe2c-ba39-4cba-83f7-3950f482bbab admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.502165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25abe2c-ba39-4cba-83f7-3950f482bbab admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.502800 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25abe2c-ba39-4cba-83f7-3950f482bbab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.504719 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25abe2c-ba39-4cba-83f7-3950f482bbab admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.506990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69d208f-c3cf-4a4c-83ec-fa7b2bfcab3d None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.510678 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f422eb-5ddc-4039-a60e-c0ad2e0716ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.511371 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f422eb-5ddc-4039-a60e-c0ad2e0716ae service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.511844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f422eb-5ddc-4039-a60e-c0ad2e0716ae service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.512155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f422eb-5ddc-4039-a60e-c0ad2e0716ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.521981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f422eb-5ddc-4039-a60e-c0ad2e0716ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.523774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f422eb-5ddc-4039-a60e-c0ad2e0716ae service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.532789 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1243/2459] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:35 2022] POST /identity/v3/projects => generated 375 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:35.536837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25abe2c-ba39-4cba-83f7-3950f482bbab admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.538690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1217/2460] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:35.604540 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25abe2c-ba39-4cba-83f7-3950f482bbab admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.610787 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d5847-304d-4aee-acee-0d6bad01ac46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.611643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d5847-304d-4aee-acee-0d6bad01ac46 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.612021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d5847-304d-4aee-acee-0d6bad01ac46 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.612412 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d5847-304d-4aee-acee-0d6bad01ac46 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.613139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d5847-304d-4aee-acee-0d6bad01ac46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.615170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d5847-304d-4aee-acee-0d6bad01ac46 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.637795 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1244/2461] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:35 2022] POST /identity/v3/users => generated 294 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:35.642138 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f422eb-5ddc-4039-a60e-c0ad2e0716ae service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.653231 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f422eb-5ddc-4039-a60e-c0ad2e0716ae service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.658043 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec76e90-d071-40c8-a63f-2dc5083559e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.658633 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec76e90-d071-40c8-a63f-2dc5083559e2 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.658914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec76e90-d071-40c8-a63f-2dc5083559e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.659192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec76e90-d071-40c8-a63f-2dc5083559e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.659835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec76e90-d071-40c8-a63f-2dc5083559e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.661475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec76e90-d071-40c8-a63f-2dc5083559e2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.672628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1218/2462] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:35.677258 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d5847-304d-4aee-acee-0d6bad01ac46 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.688444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d5847-304d-4aee-acee-0d6bad01ac46 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.696219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.696879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.697242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.697524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] PATH_INFO: `/v3/projects/f3cb30dd5ba2443fba6435cd55fee796/users/8fd53848d86841a68b6ebe347de0c0a3/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.710648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:35.715360 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3cb30dd5ba2443fba6435cd55fee796, user_id=8fd53848d86841a68b6ebe347de0c0a3, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.718937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.734602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1245/2463] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:35 2022] PUT /identity/v3/projects/f3cb30dd5ba2443fba6435cd55fee796/users/8fd53848d86841a68b6ebe347de0c0a3/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:35.738518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec76e90-d071-40c8-a63f-2dc5083559e2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.749565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec76e90-d071-40c8-a63f-2dc5083559e2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.822394 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92b6f89-51c1-428b-affa-65bd26b00981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.823071 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92b6f89-51c1-428b-affa-65bd26b00981 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.823352 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92b6f89-51c1-428b-affa-65bd26b00981 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.823626 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92b6f89-51c1-428b-affa-65bd26b00981 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.826082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92b6f89-51c1-428b-affa-65bd26b00981 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.827908 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92b6f89-51c1-428b-affa-65bd26b00981 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.831060 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.833380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:35.844958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1219/2464] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:35.845835 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92b6f89-51c1-428b-affa-65bd26b00981 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:35.849567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029c136c-f0c4-4ec5-bdb8-ade863f3906f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.856221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.856928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.857306 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.857590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] PATH_INFO: `/v3/projects/eb933a8adba1470ab329e6e6fae57d0b` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.858602 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92b6f89-51c1-428b-affa-65bd26b00981 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:35.859690 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb933a8adba1470ab329e6e6fae57d0b)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.861936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.864355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857261a2-b19b-4728-a68f-eb8a6df3580b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:35.865565 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857261a2-b19b-4728-a68f-eb8a6df3580b admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.865770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857261a2-b19b-4728-a68f-eb8a6df3580b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.866087 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857261a2-b19b-4728-a68f-eb8a6df3580b admin admin] PATH_INFO: `/v3/projects/f3cb30dd5ba2443fba6435cd55fee796/users/8fd53848d86841a68b6ebe347de0c0a3/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.868309 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857261a2-b19b-4728-a68f-eb8a6df3580b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3cb30dd5ba2443fba6435cd55fee796, user_id=8fd53848d86841a68b6ebe347de0c0a3, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:35.872805 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857261a2-b19b-4728-a68f-eb8a6df3580b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:35.885122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1220/2465] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:35 2022] PUT /identity/v3/projects/f3cb30dd5ba2443fba6435cd55fee796/users/8fd53848d86841a68b6ebe347de0c0a3/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:35.895372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b391d8-7016-41b8-a3e3-196504779f7b None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.895649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b391d8-7016-41b8-a3e3-196504779f7b None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.895957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b391d8-7016-41b8-a3e3-196504779f7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.917086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb933a8adba1470ab329e6e6fae57d0b', 'request_id': 'req-8c461766-ae27-4262-9432-8a13974ac5db'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.924792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb933a8adba1470ab329e6e6fae57d0b', 'request_id': 'req-8c461766-ae27-4262-9432-8a13974ac5db'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.928259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-95b391d8-7016-41b8-a3e3-196504779f7b None None] MFA Rules not processed for user `8fd53848d86841a68b6ebe347de0c0a3`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:35.929513 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-95b391d8-7016-41b8-a3e3-196504779f7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:35.932857 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb933a8adba1470ab329e6e6fae57d0b', 'request_id': 'req-8c461766-ae27-4262-9432-8a13974ac5db'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.945470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] The token cache is being invalidate because project eb933a8adba1470ab329e6e6fae57d0b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:35.945757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.946522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-8c461766-ae27-4262-9432-8a13974ac5db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:35.948254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1246/2466] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:35 2022] DELETE /identity/v3/projects/eb933a8adba1470ab329e6e6fae57d0b => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:35.953347 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207dc16-7f86-47fb-a767-401ec0a95dae None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:35.953734 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207dc16-7f86-47fb-a767-401ec0a95dae None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:35.954013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207dc16-7f86-47fb-a767-401ec0a95dae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:35.976213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-8207dc16-7f86-47fb-a767-401ec0a95dae None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:35.976457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:35.976879 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8207dc16-7f86-47fb-a767-401ec0a95dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:36.016319 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1221/2467] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:35 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:36.027573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b391d8-7016-41b8-a3e3-196504779f7b None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:36.029334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-95b391d8-7016-41b8-a3e3-196504779f7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:36.049566 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1247/2468] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:36.057636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4783d2eb-eaff-4515-bb3c-538a003dfad8 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.057995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4783d2eb-eaff-4515-bb3c-538a003dfad8 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.058959 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4783d2eb-eaff-4515-bb3c-538a003dfad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.080946 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4783d2eb-eaff-4515-bb3c-538a003dfad8 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:36.082381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4783d2eb-eaff-4515-bb3c-538a003dfad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:36.091265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1248/2469] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:36.095417 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4783d2eb-eaff-4515-bb3c-538a003dfad8 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:36.098520 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b391d8-7016-41b8-a3e3-196504779f7b None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:36.102230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:36.103049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.103429 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.103818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.106572 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:36.107337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4783d2eb-eaff-4515-bb3c-538a003dfad8 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:36.119641 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:36.121629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.122241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.124493 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.125182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:36.125981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:36.127611 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:36.141816 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1249/2470] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 09:02:36.146567 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:36.161158 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9541ebca-3392-4ec6-a853-3cbce8100ada admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:36.167843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64815041-9ba3-43b6-a553-8efd20bcd77f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:36.168524 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64815041-9ba3-43b6-a553-8efd20bcd77f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.168764 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64815041-9ba3-43b6-a553-8efd20bcd77f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.169038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64815041-9ba3-43b6-a553-8efd20bcd77f admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.169674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64815041-9ba3-43b6-a553-8efd20bcd77f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:36.171546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64815041-9ba3-43b6-a553-8efd20bcd77f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:36.180304 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:36.181898 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:36.197049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1222/2471] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:36 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:36.206209 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1250/2472] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:36 2022] POST /identity/v3/projects => generated 385 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:36.211121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:36.225479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd014dd7-4f11-4cad-aaf2-dbdf525f60db service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:36.303367 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dca948-c723-49aa-bd2f-72a85105e90e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:36.303981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dca948-c723-49aa-bd2f-72a85105e90e admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.304298 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dca948-c723-49aa-bd2f-72a85105e90e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.304581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dca948-c723-49aa-bd2f-72a85105e90e admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.305212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dca948-c723-49aa-bd2f-72a85105e90e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:36.307302 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dca948-c723-49aa-bd2f-72a85105e90e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:36.331431 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1223/2473] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:36 2022] POST /identity/v3/users => generated 293 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:36.344786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64815041-9ba3-43b6-a553-8efd20bcd77f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:36.356772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64815041-9ba3-43b6-a553-8efd20bcd77f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:36.361966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde4669-c170-41d4-a7fe-be242513a9e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:36.362601 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde4669-c170-41d4-a7fe-be242513a9e6 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.362969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde4669-c170-41d4-a7fe-be242513a9e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.363141 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde4669-c170-41d4-a7fe-be242513a9e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.363783 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dde4669-c170-41d4-a7fe-be242513a9e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:36.365435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dde4669-c170-41d4-a7fe-be242513a9e6 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:36.378164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1251/2474] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:36.382494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dca948-c723-49aa-bd2f-72a85105e90e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:36.393936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dca948-c723-49aa-bd2f-72a85105e90e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:36.399584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9b8155-4f25-4237-9b8f-f857f9cb58bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:36.400361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b8155-4f25-4237-9b8f-f857f9cb58bd admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.401030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b8155-4f25-4237-9b8f-f857f9cb58bd admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.401030 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b8155-4f25-4237-9b8f-f857f9cb58bd admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.401769 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9b8155-4f25-4237-9b8f-f857f9cb58bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:36.404423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9b8155-4f25-4237-9b8f-f857f9cb58bd admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:36.420344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1224/2475] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:36.422607 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde4669-c170-41d4-a7fe-be242513a9e6 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:36.441587 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde4669-c170-41d4-a7fe-be242513a9e6 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:36.446616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:36.447279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.447562 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.447835 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] PATH_INFO: `/v3/projects/91deaad648294753918206e0a1e79982/users/fafbfea000024b7ca5b9b38529830239/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.468629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91deaad648294753918206e0a1e79982, user_id=fafbfea000024b7ca5b9b38529830239, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:36.468629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:36.478658 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:36.489447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1252/2476] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:36 2022] PUT /identity/v3/projects/91deaad648294753918206e0a1e79982/users/fafbfea000024b7ca5b9b38529830239/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:36.502307 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f41211-7e64-44c1-8ee1-061d0f57e558 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.502495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f41211-7e64-44c1-8ee1-061d0f57e558 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.502843 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f41211-7e64-44c1-8ee1-061d0f57e558 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.542384 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:36.555020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b0f41211-7e64-44c1-8ee1-061d0f57e558 None None] MFA Rules not processed for user `fafbfea000024b7ca5b9b38529830239`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:36.557212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b0f41211-7e64-44c1-8ee1-061d0f57e558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:36.670620 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1225/2477] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:36.677374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:36.679177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:36.755031 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32858f97-3514-4a07-8630-8ac07af09cf5 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:36.758636 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bf596f-37e0-4a86-9b7d-bb7d0716ba72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:36.759240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf596f-37e0-4a86-9b7d-bb7d0716ba72 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:36.759520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf596f-37e0-4a86-9b7d-bb7d0716ba72 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:36.759790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf596f-37e0-4a86-9b7d-bb7d0716ba72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:36.770485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bf596f-37e0-4a86-9b7d-bb7d0716ba72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:36.771998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bf596f-37e0-4a86-9b7d-bb7d0716ba72 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:36.783146 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1253/2478] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:43.030312 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f41211-7e64-44c1-8ee1-061d0f57e558 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:43.040153 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f41211-7e64-44c1-8ee1-061d0f57e558 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:43.044337 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68206f9a-ead6-40aa-9011-cc2a43941d13 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:43.044925 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68206f9a-ead6-40aa-9011-cc2a43941d13 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:43.045201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68206f9a-ead6-40aa-9011-cc2a43941d13 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:43.045496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68206f9a-ead6-40aa-9011-cc2a43941d13 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:43.105132 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68206f9a-ead6-40aa-9011-cc2a43941d13 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:43.107622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68206f9a-ead6-40aa-9011-cc2a43941d13 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:43.117819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1226/2479] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:43 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:46.776970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=27 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:53.112782 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=58 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:53.660069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bf596f-37e0-4a86-9b7d-bb7d0716ba72 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:53.722333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bf596f-37e0-4a86-9b7d-bb7d0716ba72 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:53.726808 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:53.727451 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:53.727728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:53.728002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] PATH_INFO: `/v3/users/fafbfea000024b7ca5b9b38529830239` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:53.729117 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fafbfea000024b7ca5b9b38529830239)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:53.731139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:53.778234 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fafbfea000024b7ca5b9b38529830239'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:53.785400 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fafbfea000024b7ca5b9b38529830239'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:53.797302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fafbfea000024b7ca5b9b38529830239'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:53.798037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fafbfea000024b7ca5b9b38529830239'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:53.798804 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fafbfea000024b7ca5b9b38529830239'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:53.812510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1254/2480] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:53 2022] DELETE /identity/v3/users/fafbfea000024b7ca5b9b38529830239 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:53.817257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93f86d3-3d87-4bef-b839-aa64ebb0bb76 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:53.817546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93f86d3-3d87-4bef-b839-aa64ebb0bb76 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:53.817824 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93f86d3-3d87-4bef-b839-aa64ebb0bb76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:53.837779 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-c93f86d3-3d87-4bef-b839-aa64ebb0bb76 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:53.838988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c93f86d3-3d87-4bef-b839-aa64ebb0bb76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:53.898031 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1227/2481] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:53.908432 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:53.909803 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:53.967330 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36206c35-cd58-4d11-9a07-e4c071045974 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:53.970975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fa3d11-9ab7-4bac-9360-c3349db20280 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:53.971577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa3d11-9ab7-4bac-9360-c3349db20280 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:53.971854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa3d11-9ab7-4bac-9360-c3349db20280 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:53.972125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa3d11-9ab7-4bac-9360-c3349db20280 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:53.981709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fa3d11-9ab7-4bac-9360-c3349db20280 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:53.983144 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fa3d11-9ab7-4bac-9360-c3349db20280 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:53.993357 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1255/2482] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:54.293652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93f86d3-3d87-4bef-b839-aa64ebb0bb76 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:54.294729 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-c93f86d3-3d87-4bef-b839-aa64ebb0bb76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:54.310535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93f86d3-3d87-4bef-b839-aa64ebb0bb76 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:54.316632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:54.317257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.317531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.317802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] PATH_INFO: `/v3/projects/91deaad648294753918206e0a1e79982` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.320113 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91deaad648294753918206e0a1e79982)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:54.323615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:54.383167 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91deaad648294753918206e0a1e79982', 'request_id': 'req-3802d5d0-b574-4b01-b152-9ef11dd55f67'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:54.391234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91deaad648294753918206e0a1e79982', 'request_id': 'req-3802d5d0-b574-4b01-b152-9ef11dd55f67'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:54.399115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91deaad648294753918206e0a1e79982', 'request_id': 'req-3802d5d0-b574-4b01-b152-9ef11dd55f67'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:54.414220 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] The token cache is being invalidate because project 91deaad648294753918206e0a1e79982 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:54.414503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:54.415122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3802d5d0-b574-4b01-b152-9ef11dd55f67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:54.417218 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1228/2483] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:54 2022] DELETE /identity/v3/projects/91deaad648294753918206e0a1e79982 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:54.422290 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9171e7-a3e1-4230-b338-9482dcf24ce2 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.422659 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9171e7-a3e1-4230-b338-9482dcf24ce2 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.422955 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9171e7-a3e1-4230-b338-9482dcf24ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.444353 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-8d9171e7-a3e1-4230-b338-9482dcf24ce2 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:54.445597 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-8d9171e7-a3e1-4230-b338-9482dcf24ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:54.464285 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:54.519846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1256/2484] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:54.527212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bea3e1-d70f-45fd-b06e-adbe32665a32 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.527503 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bea3e1-d70f-45fd-b06e-adbe32665a32 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.527781 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bea3e1-d70f-45fd-b06e-adbe32665a32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.548001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-62bea3e1-d70f-45fd-b06e-adbe32665a32 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:54.549350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-62bea3e1-d70f-45fd-b06e-adbe32665a32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:54.557234 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1229/2485] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:54.562168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9171e7-a3e1-4230-b338-9482dcf24ce2 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:54.571704 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9171e7-a3e1-4230-b338-9482dcf24ce2 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:54.577037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:54.577634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.577913 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.578181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] PATH_INFO: `/v3/domains` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.578818 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93080) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:54.579122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:54.580733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:54.592486 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1257/2486] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 24 09:02:54.596829 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bea3e1-d70f-45fd-b06e-adbe32665a32 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:54.607037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bea3e1-d70f-45fd-b06e-adbe32665a32 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:54.611719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd57ec3e-593a-4932-beb2-f6dcb43273c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:54.612354 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57ec3e-593a-4932-beb2-f6dcb43273c4 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.612637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57ec3e-593a-4932-beb2-f6dcb43273c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.612912 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57ec3e-593a-4932-beb2-f6dcb43273c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.613557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd57ec3e-593a-4932-beb2-f6dcb43273c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:54.615486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd57ec3e-593a-4932-beb2-f6dcb43273c4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:54.644057 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1230/2487] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:54 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:54.648338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:54.658712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82214282-f72a-4418-aad3-f1bb5b67dd19 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:54.715939 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b259052c-0d69-452c-a573-c85ea43cbc06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:54.716629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b259052c-0d69-452c-a573-c85ea43cbc06 admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.716849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b259052c-0d69-452c-a573-c85ea43cbc06 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.717125 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b259052c-0d69-452c-a573-c85ea43cbc06 admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.717755 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b259052c-0d69-452c-a573-c85ea43cbc06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:54.719635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b259052c-0d69-452c-a573-c85ea43cbc06 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:54.741681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1258/2488] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:54 2022] POST /identity/v3/users => generated 287 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:54.747233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd57ec3e-593a-4932-beb2-f6dcb43273c4 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:54.757442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd57ec3e-593a-4932-beb2-f6dcb43273c4 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:54.762448 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454e9a9d-f85f-4ff9-82b4-8ee91d46213f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:54.763035 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454e9a9d-f85f-4ff9-82b4-8ee91d46213f admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.763310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454e9a9d-f85f-4ff9-82b4-8ee91d46213f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.763579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454e9a9d-f85f-4ff9-82b4-8ee91d46213f admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.764259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454e9a9d-f85f-4ff9-82b4-8ee91d46213f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:54.765833 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454e9a9d-f85f-4ff9-82b4-8ee91d46213f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:54.775585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1231/2489] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:54.779782 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b259052c-0d69-452c-a573-c85ea43cbc06 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:54.789736 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b259052c-0d69-452c-a573-c85ea43cbc06 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:54.795032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db150db-5fe4-4ad0-bc19-7121340ce841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:54.795623 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db150db-5fe4-4ad0-bc19-7121340ce841 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.795901 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db150db-5fe4-4ad0-bc19-7121340ce841 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.796200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db150db-5fe4-4ad0-bc19-7121340ce841 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.796855 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db150db-5fe4-4ad0-bc19-7121340ce841 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:54.798418 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db150db-5fe4-4ad0-bc19-7121340ce841 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:54.809733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1259/2490] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:54.813903 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454e9a9d-f85f-4ff9-82b4-8ee91d46213f admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:54.823851 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454e9a9d-f85f-4ff9-82b4-8ee91d46213f admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:54.828558 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:54.829203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.829486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.829762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] PATH_INFO: `/v3/projects/d0ccb79814f5432cb21c619e974fecd8/users/f452f27fc0d34f73aa22da2a09d54eb8/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.844556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0ccb79814f5432cb21c619e974fecd8, user_id=f452f27fc0d34f73aa22da2a09d54eb8, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:54.847334 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:54.860807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1232/2491] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:54 2022] PUT /identity/v3/projects/d0ccb79814f5432cb21c619e974fecd8/users/f452f27fc0d34f73aa22da2a09d54eb8/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:54.865994 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684da0d-71f4-4494-b636-6d6cf084fa5e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:54.866375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684da0d-71f4-4494-b636-6d6cf084fa5e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:54.866708 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684da0d-71f4-4494-b636-6d6cf084fa5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:54.886694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-e684da0d-71f4-4494-b636-6d6cf084fa5e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:54.887842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e684da0d-71f4-4494-b636-6d6cf084fa5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:54.948196 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1260/2492] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:54.957650 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:54.959318 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:54.982634 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:55.021170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db477210-39d5-4b2c-95e1-54aad29c500e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:55.024848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b542977-35b1-4a32-ba03-b1bf546e32e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:55.025442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b542977-35b1-4a32-ba03-b1bf546e32e4 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:55.025719 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b542977-35b1-4a32-ba03-b1bf546e32e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:55.025987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b542977-35b1-4a32-ba03-b1bf546e32e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:55.035506 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b542977-35b1-4a32-ba03-b1bf546e32e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:55.036944 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b542977-35b1-4a32-ba03-b1bf546e32e4 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:55.046920 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1233/2493] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:55.216532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e684da0d-71f4-4494-b636-6d6cf084fa5e None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:55.217969 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e684da0d-71f4-4494-b636-6d6cf084fa5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:55.231168 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e684da0d-71f4-4494-b636-6d6cf084fa5e None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:55.235546 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:55.236208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:55.236494 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:55.236768 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] PATH_INFO: `/v3/users/de865f4d44d94a27861f396308af21af` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:55.237880 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de865f4d44d94a27861f396308af21af)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:55.239802 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:55.266755 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:55.296374 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de865f4d44d94a27861f396308af21af'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.304381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de865f4d44d94a27861f396308af21af'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.318233 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de865f4d44d94a27861f396308af21af'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.318995 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de865f4d44d94a27861f396308af21af'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.319773 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de865f4d44d94a27861f396308af21af'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.335557 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1261/2494] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:02:55 2022] DELETE /identity/v3/users/de865f4d44d94a27861f396308af21af => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:55.341246 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc83b5ae-7a03-4dbf-beac-adbdb006b516 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:55.341740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc83b5ae-7a03-4dbf-beac-adbdb006b516 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:55.341841 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc83b5ae-7a03-4dbf-beac-adbdb006b516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:55.361159 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-dc83b5ae-7a03-4dbf-beac-adbdb006b516 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:55.362359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-dc83b5ae-7a03-4dbf-beac-adbdb006b516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:55.428974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1234/2495] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:55.438726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:55.440214 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:55.521484 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e549753-84ad-4a64-b26b-32db7ea835e2 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:55.525738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:55.527022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:55.527022 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:55.528013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:55.540291 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:55.541964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:55.559059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1262/2496] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:02:55.740856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b19fb-c868-499d-a265-d20c5003ea04 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:55.741151 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b19fb-c868-499d-a265-d20c5003ea04 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:55.741872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b19fb-c868-499d-a265-d20c5003ea04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:55.782931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b43b19fb-c868-499d-a265-d20c5003ea04 None None] MFA Rules not processed for user `f452f27fc0d34f73aa22da2a09d54eb8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:55.783926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b43b19fb-c868-499d-a265-d20c5003ea04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:55.869846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:55.872235 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:55.889517 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef1e415-164e-46a9-8a06-83ef6784965a service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:55.894750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:55.895415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1235/2497] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:55 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:02:55.896830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:55.897401 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:55.897908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] PATH_INFO: `/v3/projects/9b5f5cc5e1684a8abe3ec54303fb8722` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:55.899314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b5f5cc5e1684a8abe3ec54303fb8722)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:55.901299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:55.907612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b19fb-c868-499d-a265-d20c5003ea04 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:55.909373 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b43b19fb-c868-499d-a265-d20c5003ea04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:55.963017 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b5f5cc5e1684a8abe3ec54303fb8722', 'request_id': 'req-fb9c761a-370e-4b28-ba23-14d26d4de1df'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.971675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b5f5cc5e1684a8abe3ec54303fb8722', 'request_id': 'req-fb9c761a-370e-4b28-ba23-14d26d4de1df'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.979056 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b5f5cc5e1684a8abe3ec54303fb8722', 'request_id': 'req-fb9c761a-370e-4b28-ba23-14d26d4de1df'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.989677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b19fb-c868-499d-a265-d20c5003ea04 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:55.995161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:55.996875 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] The token cache is being invalidate because project 9b5f5cc5e1684a8abe3ec54303fb8722 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:02:55.998474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:55.999173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-fb9c761a-370e-4b28-ba23-14d26d4de1df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:02:56.001059 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1263/2498] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:02:55 2022] DELETE /identity/v3/projects/9b5f5cc5e1684a8abe3ec54303fb8722 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:02:56.005332 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4191b75-7175-48a0-902c-76c07ab5876e None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.005616 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4191b75-7175-48a0-902c-76c07ab5876e None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.005893 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4191b75-7175-48a0-902c-76c07ab5876e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.026544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a4191b75-7175-48a0-902c-76c07ab5876e None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:56.028088 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a4191b75-7175-48a0-902c-76c07ab5876e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:56.046115 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:56.046726 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.047005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.047301 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.049143 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:56.049834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:56.091126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1264/2499] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:56.098918 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa92168-d390-4f5b-9f92-f16d79a9c661 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.099381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa92168-d390-4f5b-9f92-f16d79a9c661 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.099681 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa92168-d390-4f5b-9f92-f16d79a9c661 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.111062 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:56.112702 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:56.122154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4aa92168-d390-4f5b-9f92-f16d79a9c661 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:56.123371 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4aa92168-d390-4f5b-9f92-f16d79a9c661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:56.125988 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1236/2500] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:55 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:02:56.132739 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1265/2501] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:56.136901 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:56.147715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772cd941-b1ce-4039-9b77-e2564203b0a1 service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:56.155074 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:56.155687 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.155965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.156264 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.156926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:02:56.157232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:56.160210 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:56.172527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1237/2502] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:02:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:02:56.176996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa92168-d390-4f5b-9f92-f16d79a9c661 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:56.186420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa92168-d390-4f5b-9f92-f16d79a9c661 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:56.191086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c5f5e8-b904-44f5-a450-c1895876e72b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:56.191694 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c5f5e8-b904-44f5-a450-c1895876e72b admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.191972 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c5f5e8-b904-44f5-a450-c1895876e72b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.192275 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c5f5e8-b904-44f5-a450-c1895876e72b admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.192923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c5f5e8-b904-44f5-a450-c1895876e72b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:56.194801 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c5f5e8-b904-44f5-a450-c1895876e72b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:56.234120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1266/2503] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:02:56 2022] POST /identity/v3/projects => generated 399 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:02:56.235254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:56.245980 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd8615b-f29a-4912-8946-af4f7f9966e2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:56.309449 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d8af46-1585-40e5-811e-0364265cd301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:56.310059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d8af46-1585-40e5-811e-0364265cd301 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.310342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d8af46-1585-40e5-811e-0364265cd301 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.310652 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d8af46-1585-40e5-811e-0364265cd301 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.311553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d8af46-1585-40e5-811e-0364265cd301 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:56.313762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d8af46-1585-40e5-811e-0364265cd301 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:56.336267 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1238/2504] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:02:56 2022] POST /identity/v3/users => generated 300 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:02:56.341242 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c5f5e8-b904-44f5-a450-c1895876e72b admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:56.351544 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c5f5e8-b904-44f5-a450-c1895876e72b admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:56.356900 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d998d77-df91-4e0d-8c9f-0846832afbb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:56.357497 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d998d77-df91-4e0d-8c9f-0846832afbb0 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.357776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d998d77-df91-4e0d-8c9f-0846832afbb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.358045 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d998d77-df91-4e0d-8c9f-0846832afbb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.358862 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d998d77-df91-4e0d-8c9f-0846832afbb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:56.360474 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d998d77-df91-4e0d-8c9f-0846832afbb0 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:56.372047 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1267/2505] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:02:56.375887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d8af46-1585-40e5-811e-0364265cd301 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:56.387014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d8af46-1585-40e5-811e-0364265cd301 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:56.391970 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b7e8a9-82e9-4439-a01c-6a92175358a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:56.392627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7e8a9-82e9-4439-a01c-6a92175358a7 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.392913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7e8a9-82e9-4439-a01c-6a92175358a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.393215 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7e8a9-82e9-4439-a01c-6a92175358a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.393913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b7e8a9-82e9-4439-a01c-6a92175358a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:56.395467 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b7e8a9-82e9-4439-a01c-6a92175358a7 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:56.425649 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1239/2506] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:02:56 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:02:56.429555 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d998d77-df91-4e0d-8c9f-0846832afbb0 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:56.440510 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d998d77-df91-4e0d-8c9f-0846832afbb0 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:56.445401 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:56.446065 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.446370 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.446642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] PATH_INFO: `/v3/projects/b530d7784c9d4c5a980aa4d42edaac34/users/bd476bc38809482f87b186724f385361/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.474154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] RBAC: Authorizing `identity:create_grant(project_id=b530d7784c9d4c5a980aa4d42edaac34, user_id=bd476bc38809482f87b186724f385361, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:56.474154 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:56.482830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1268/2507] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:02:56 2022] PUT /identity/v3/projects/b530d7784c9d4c5a980aa4d42edaac34/users/bd476bc38809482f87b186724f385361/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:02:56.549810 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e254e8ee-af1c-4331-80f8-fe74c19a9699 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.550131 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e254e8ee-af1c-4331-80f8-fe74c19a9699 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.550447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e254e8ee-af1c-4331-80f8-fe74c19a9699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.579510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e254e8ee-af1c-4331-80f8-fe74c19a9699 None None] MFA Rules not processed for user `bd476bc38809482f87b186724f385361`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:02:56.580748 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e254e8ee-af1c-4331-80f8-fe74c19a9699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:56.597768 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:02:56.645865 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1240/2508] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:02:56 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:02:56.658247 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:02:56.660048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:02:56.715963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889b1808-fba4-40e5-93c4-0c92792fdcca admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:02:56.719629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:02:56.720254 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:02:56.720564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:02:56.720837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:02:56.729097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:02:56.731082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:02:56.739996 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1269/2509] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:02:56 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:00.705932 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c9423d-4d0f-40b8-b3cf-1fb7e8a40c45 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:00.706390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c9423d-4d0f-40b8-b3cf-1fb7e8a40c45 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:00.706597 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c9423d-4d0f-40b8-b3cf-1fb7e8a40c45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:00.726049 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-b2c9423d-4d0f-40b8-b3cf-1fb7e8a40c45 None None] MFA Rules not processed for user `f452f27fc0d34f73aa22da2a09d54eb8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:00.727265 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b2c9423d-4d0f-40b8-b3cf-1fb7e8a40c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:00.796478 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1241/2510] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:01.050819 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:01.052337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:01.110135 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344128f9-7b1d-4e11-a417-0f5731b0fbf1 service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:01.113841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:01.114444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:01.114750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:01.115021 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:01.116938 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:01.128324 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:01.129809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:01.139164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1270/2511] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:01 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:01.840226 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c9423d-4d0f-40b8-b3cf-1fb7e8a40c45 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:01.850421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c9423d-4d0f-40b8-b3cf-1fb7e8a40c45 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:01.853991 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:01.854590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:01.854895 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:01.855170 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:01.857001 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:01.922052 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:01.923907 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:01.938872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1242/2512] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:01 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:03.636075 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:03.639345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:03.656378 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:03.779744 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695aadc2-9141-406e-b7f5-5595d2c728e2 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:03.786149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b799e94-e253-4539-a247-2be44e9c04b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:03.787190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b799e94-e253-4539-a247-2be44e9c04b7 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:03.787190 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b799e94-e253-4539-a247-2be44e9c04b7 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:03.787688 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b799e94-e253-4539-a247-2be44e9c04b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:03.789638 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-7b799e94-e253-4539-a247-2be44e9c04b7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:03.812426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b799e94-e253-4539-a247-2be44e9c04b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:03.814086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b799e94-e253-4539-a247-2be44e9c04b7 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:03.831868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1271/2513] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:11.930426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=74 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:13.224931 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:13.292422 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76638b0e-ee40-4b17-967f-d05b5bf1ffaa service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:13.297378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:13.298028 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:13.298310 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:13.298584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] PATH_INFO: `/v3/users/bd476bc38809482f87b186724f385361` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:13.299694 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd476bc38809482f87b186724f385361)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:13.301643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:13.352845 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd476bc38809482f87b186724f385361'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.360526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd476bc38809482f87b186724f385361'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.373599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd476bc38809482f87b186724f385361'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.374290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd476bc38809482f87b186724f385361'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.374899 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd476bc38809482f87b186724f385361'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.387927 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1243/2514] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:03:13 2022] DELETE /identity/v3/users/bd476bc38809482f87b186724f385361 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:13.392702 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb4c7e-5956-49e6-86ac-9f75c5bee006 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:13.393219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb4c7e-5956-49e6-86ac-9f75c5bee006 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:13.393319 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb4c7e-5956-49e6-86ac-9f75c5bee006 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:13.412679 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-4acb4c7e-5956-49e6-86ac-9f75c5bee006 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:13.413884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4acb4c7e-5956-49e6-86ac-9f75c5bee006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:13.473338 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1272/2515] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:13.483601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:13.485054 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:13.542157 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28445ee-24a9-4351-b51c-2f1c6b691da5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:13.546839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:13.547442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:13.547708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:13.547977 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:13.556930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:13.558335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:13.568554 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1244/2516] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:13.851097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acb4c7e-5956-49e6-86ac-9f75c5bee006 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:13.852662 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4acb4c7e-5956-49e6-86ac-9f75c5bee006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:13.867182 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acb4c7e-5956-49e6-86ac-9f75c5bee006 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:13.874025 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:13.874999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:13.875435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:13.875884 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] PATH_INFO: `/v3/projects/b530d7784c9d4c5a980aa4d42edaac34` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:13.877276 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b530d7784c9d4c5a980aa4d42edaac34)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:13.879872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:13.935335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b530d7784c9d4c5a980aa4d42edaac34', 'request_id': 'req-d7f5487d-68da-44dd-a468-470e461b521b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.943241 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b530d7784c9d4c5a980aa4d42edaac34', 'request_id': 'req-d7f5487d-68da-44dd-a468-470e461b521b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.952877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b530d7784c9d4c5a980aa4d42edaac34', 'request_id': 'req-d7f5487d-68da-44dd-a468-470e461b521b'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.969693 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] The token cache is being invalidate because project b530d7784c9d4c5a980aa4d42edaac34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:03:13.969988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.970725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-d7f5487d-68da-44dd-a468-470e461b521b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:13.972329 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1273/2517] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:03:13 2022] DELETE /identity/v3/projects/b530d7784c9d4c5a980aa4d42edaac34 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:03:23.013366 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:23.014818 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:23.027360 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87236720-67e5-458a-b1bf-92389029fa59 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:23.045322 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:23.085622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:23.086256 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:23.086536 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:23.086806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] PATH_INFO: `/v3/users/461a9ee13a924592ab16c58f36c0aff5` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:23.087984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] RBAC: Authorizing `identity:delete_user(user_id=461a9ee13a924592ab16c58f36c0aff5)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:23.090050 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:23.140987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '461a9ee13a924592ab16c58f36c0aff5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.147965 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '461a9ee13a924592ab16c58f36c0aff5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.159982 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '461a9ee13a924592ab16c58f36c0aff5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.160754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '461a9ee13a924592ab16c58f36c0aff5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.161372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '461a9ee13a924592ab16c58f36c0aff5'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.174711 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1245/2518] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:03:23 2022] DELETE /identity/v3/users/461a9ee13a924592ab16c58f36c0aff5 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:03:23.179170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4e160-b857-4bb1-b3c0-97debc47733f None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:23.179669 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4e160-b857-4bb1-b3c0-97debc47733f None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:23.179820 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4e160-b857-4bb1-b3c0-97debc47733f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:23.200215 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-75e4e160-b857-4bb1-b3c0-97debc47733f None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:23.201434 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-75e4e160-b857-4bb1-b3c0-97debc47733f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:23.213817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:23.265750 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1274/2519] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:23.276203 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:23.277623 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:23.335593 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a1c769-e738-4da7-9a2b-159552e0680e admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:23.339208 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9052b5-f688-493a-aaf1-15debf3e21a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:23.339844 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9052b5-f688-493a-aaf1-15debf3e21a9 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:23.340127 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9052b5-f688-493a-aaf1-15debf3e21a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:23.340426 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9052b5-f688-493a-aaf1-15debf3e21a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:23.349573 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9052b5-f688-493a-aaf1-15debf3e21a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:23.351000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9052b5-f688-493a-aaf1-15debf3e21a9 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:23.361388 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1246/2520] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:23.659526 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4e160-b857-4bb1-b3c0-97debc47733f None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:23.661121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-75e4e160-b857-4bb1-b3c0-97debc47733f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:23.678231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4e160-b857-4bb1-b3c0-97debc47733f None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:23.684265 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:23.685710 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:23.686057 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:23.686396 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] PATH_INFO: `/v3/projects/ff7c2d77b80e4571b5dbec9402bbe154` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:23.687627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff7c2d77b80e4571b5dbec9402bbe154)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:23.689963 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:23.741671 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff7c2d77b80e4571b5dbec9402bbe154', 'request_id': 'req-e008d443-d4e2-48da-86ec-8c7604251087'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.749817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff7c2d77b80e4571b5dbec9402bbe154', 'request_id': 'req-e008d443-d4e2-48da-86ec-8c7604251087'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.758649 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff7c2d77b80e4571b5dbec9402bbe154', 'request_id': 'req-e008d443-d4e2-48da-86ec-8c7604251087'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.769696 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] The token cache is being invalidate because project ff7c2d77b80e4571b5dbec9402bbe154 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:03:23.769990 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.770626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.772118 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1275/2521] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:03:23 2022] DELETE /identity/v3/projects/ff7c2d77b80e4571b5dbec9402bbe154 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:23.776274 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9052b5-f688-493a-aaf1-15debf3e21a9 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:23.786674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9052b5-f688-493a-aaf1-15debf3e21a9 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:23.829823 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:23.848760 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:23.849423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:23.849715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:23.849990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] PATH_INFO: `/v3/users/8fd53848d86841a68b6ebe347de0c0a3` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:23.851188 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fd53848d86841a68b6ebe347de0c0a3)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:23.853198 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:23.908734 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fd53848d86841a68b6ebe347de0c0a3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.916406 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fd53848d86841a68b6ebe347de0c0a3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.933612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fd53848d86841a68b6ebe347de0c0a3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.934580 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fd53848d86841a68b6ebe347de0c0a3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.935577 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-60c5151c-5885-494d-960c-7dadd2b792b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fd53848d86841a68b6ebe347de0c0a3'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:23.962871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1247/2522] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:03:23 2022] DELETE /identity/v3/users/8fd53848d86841a68b6ebe347de0c0a3 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:24.339445 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:24.342614 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:24.361709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008d443-d4e2-48da-86ec-8c7604251087 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:24.444726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:24.445460 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:24.445847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:24.446126 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] PATH_INFO: `/v3/projects/f3cb30dd5ba2443fba6435cd55fee796` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:24.447349 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3cb30dd5ba2443fba6435cd55fee796)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:24.449889 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:24.508612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:24.513300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3cb30dd5ba2443fba6435cd55fee796', 'request_id': 'req-76c7269d-0890-4a33-b1c7-eb61a4559807'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:24.522211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3cb30dd5ba2443fba6435cd55fee796', 'request_id': 'req-76c7269d-0890-4a33-b1c7-eb61a4559807'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:24.535709 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3cb30dd5ba2443fba6435cd55fee796', 'request_id': 'req-76c7269d-0890-4a33-b1c7-eb61a4559807'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:24.551626 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] The token cache is being invalidate because project f3cb30dd5ba2443fba6435cd55fee796 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:03:24.552345 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:24.553176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-76c7269d-0890-4a33-b1c7-eb61a4559807 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:24.558061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1276/2523] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:03:24 2022] DELETE /identity/v3/projects/f3cb30dd5ba2443fba6435cd55fee796 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:24.560152 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b59de7-0c06-4618-9f5b-5c2b34e6d961 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:24.560837 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b59de7-0c06-4618-9f5b-5c2b34e6d961 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:24.561121 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b59de7-0c06-4618-9f5b-5c2b34e6d961 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:24.587612 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-94b59de7-0c06-4618-9f5b-5c2b34e6d961 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:24.589300 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-94b59de7-0c06-4618-9f5b-5c2b34e6d961 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:24.673463 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1248/2524] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:24.680953 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905892d-372a-49de-8d27-e08fd1fce579 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:24.681239 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905892d-372a-49de-8d27-e08fd1fce579 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:24.681521 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905892d-372a-49de-8d27-e08fd1fce579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:24.702147 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0905892d-372a-49de-8d27-e08fd1fce579 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:24.703358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0905892d-372a-49de-8d27-e08fd1fce579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:24.711476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1277/2525] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:24.716439 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b59de7-0c06-4618-9f5b-5c2b34e6d961 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:24.726361 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b59de7-0c06-4618-9f5b-5c2b34e6d961 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:24.731282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:24.731883 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:24.732199 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:24.732475 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:24.733111 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:03:24.733415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:24.735003 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:24.748331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1249/2526] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:03:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:03:24.752840 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0905892d-372a-49de-8d27-e08fd1fce579 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:24.763245 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0905892d-372a-49de-8d27-e08fd1fce579 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:24.768923 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638ed3b0-5400-449b-96e6-1c110d9dcdfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:24.769627 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ed3b0-5400-449b-96e6-1c110d9dcdfe admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:24.769980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ed3b0-5400-449b-96e6-1c110d9dcdfe admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:24.770253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ed3b0-5400-449b-96e6-1c110d9dcdfe admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:24.770887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638ed3b0-5400-449b-96e6-1c110d9dcdfe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:24.772979 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638ed3b0-5400-449b-96e6-1c110d9dcdfe admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:24.811387 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1278/2527] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:03:24 2022] POST /identity/v3/projects => generated 383 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:24.812628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:24.823728 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65635382-7464-4a81-8161-35b5215e3e46 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:24.883349 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:24.907708 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec39a3-b106-4411-980c-db4a0289c1b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:24.929032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec39a3-b106-4411-980c-db4a0289c1b2 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:24.929032 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec39a3-b106-4411-980c-db4a0289c1b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:24.929359 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec39a3-b106-4411-980c-db4a0289c1b2 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:24.931930 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ec39a3-b106-4411-980c-db4a0289c1b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:24.936617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ec39a3-b106-4411-980c-db4a0289c1b2 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:24.981486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1250/2528] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:03:24 2022] POST /identity/v3/users => generated 292 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:24.986377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638ed3b0-5400-449b-96e6-1c110d9dcdfe admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.013483 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638ed3b0-5400-449b-96e6-1c110d9dcdfe admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.019590 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4da34b-6b41-402a-a70b-5659107ba37d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.020525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4da34b-6b41-402a-a70b-5659107ba37d admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.020908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4da34b-6b41-402a-a70b-5659107ba37d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.021701 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4da34b-6b41-402a-a70b-5659107ba37d admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.022325 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4da34b-6b41-402a-a70b-5659107ba37d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.025927 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4da34b-6b41-402a-a70b-5659107ba37d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.043331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1279/2529] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:24 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:03:25.045308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec39a3-b106-4411-980c-db4a0289c1b2 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.063518 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec39a3-b106-4411-980c-db4a0289c1b2 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.078905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c130cd21-a700-4baf-99f0-bb2d6452f624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.079510 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c130cd21-a700-4baf-99f0-bb2d6452f624 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.079789 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c130cd21-a700-4baf-99f0-bb2d6452f624 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.080063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c130cd21-a700-4baf-99f0-bb2d6452f624 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.080740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c130cd21-a700-4baf-99f0-bb2d6452f624 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.082331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c130cd21-a700-4baf-99f0-bb2d6452f624 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.110290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1251/2530] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:25 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:03:25.113817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4da34b-6b41-402a-a70b-5659107ba37d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.129754 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4da34b-6b41-402a-a70b-5659107ba37d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.136377 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012f6ea6-9d8d-4006-a886-68035fd6598f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.137257 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012f6ea6-9d8d-4006-a886-68035fd6598f admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.137608 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012f6ea6-9d8d-4006-a886-68035fd6598f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.137882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012f6ea6-9d8d-4006-a886-68035fd6598f admin admin] PATH_INFO: `/v3/projects/5226b786e04e4b6985bed73fa3322e6e/users/8497fed43a7746d28e288c64d18dc4a6/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.162185 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012f6ea6-9d8d-4006-a886-68035fd6598f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5226b786e04e4b6985bed73fa3322e6e, user_id=8497fed43a7746d28e288c64d18dc4a6, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.165707 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012f6ea6-9d8d-4006-a886-68035fd6598f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.183846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1280/2531] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:03:25 2022] PUT /identity/v3/projects/5226b786e04e4b6985bed73fa3322e6e/users/8497fed43a7746d28e288c64d18dc4a6/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:03:25.195206 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665ce42f-9797-47a2-9615-8a42979757a2 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.195494 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665ce42f-9797-47a2-9615-8a42979757a2 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.197055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665ce42f-9797-47a2-9615-8a42979757a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.235857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-665ce42f-9797-47a2-9615-8a42979757a2 None None] MFA Rules not processed for user `8497fed43a7746d28e288c64d18dc4a6`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:25.237096 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-665ce42f-9797-47a2-9615-8a42979757a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:25.329908 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a2781-a612-4728-b0ab-110c3ea0cdbd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.330195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a2781-a612-4728-b0ab-110c3ea0cdbd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.330468 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a2781-a612-4728-b0ab-110c3ea0cdbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.343020 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1252/2532] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:25.349421 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665ce42f-9797-47a2-9615-8a42979757a2 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.355119 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-422a2781-a612-4728-b0ab-110c3ea0cdbd None None] MFA Rules not processed for user `3cddd27c37514e74a38430c46cef16fa`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:25.356596 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-422a2781-a612-4728-b0ab-110c3ea0cdbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:25.423290 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665ce42f-9797-47a2-9615-8a42979757a2 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.424166 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1281/2533] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:25.432214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466aaaa1-48a8-45ff-80c6-b4c16aeb832b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.432826 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466aaaa1-48a8-45ff-80c6-b4c16aeb832b admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.433104 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466aaaa1-48a8-45ff-80c6-b4c16aeb832b admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.433377 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466aaaa1-48a8-45ff-80c6-b4c16aeb832b admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.434846 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466aaaa1-48a8-45ff-80c6-b4c16aeb832b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.436787 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466aaaa1-48a8-45ff-80c6-b4c16aeb832b admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.465296 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1253/2534] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:03:25 2022] POST /identity/v3/projects => generated 387 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:25.469269 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a2781-a612-4728-b0ab-110c3ea0cdbd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.480038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a2781-a612-4728-b0ab-110c3ea0cdbd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.538978 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9a4d67-1eea-4f50-ab47-c57d46b60b1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.539505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9a4d67-1eea-4f50-ab47-c57d46b60b1c admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.539780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9a4d67-1eea-4f50-ab47-c57d46b60b1c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.540048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9a4d67-1eea-4f50-ab47-c57d46b60b1c admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.540717 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9a4d67-1eea-4f50-ab47-c57d46b60b1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.542602 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9a4d67-1eea-4f50-ab47-c57d46b60b1c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.568449 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1282/2535] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:03:25 2022] POST /identity/v3/users => generated 300 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:25.569691 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466aaaa1-48a8-45ff-80c6-b4c16aeb832b admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.580863 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466aaaa1-48a8-45ff-80c6-b4c16aeb832b admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.585523 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345c94e6-e972-408b-b032-c97366c709ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.586114 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345c94e6-e972-408b-b032-c97366c709ac admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.586390 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345c94e6-e972-408b-b032-c97366c709ac admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.586661 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345c94e6-e972-408b-b032-c97366c709ac admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.587295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345c94e6-e972-408b-b032-c97366c709ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.588916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345c94e6-e972-408b-b032-c97366c709ac admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.599308 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1254/2536] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:03:25.603256 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9a4d67-1eea-4f50-ab47-c57d46b60b1c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.608886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=90,keystone:DELETE=4,keystone:INSERT=6 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:25.615176 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9a4d67-1eea-4f50-ab47-c57d46b60b1c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.621695 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b579d6-2a88-4c63-9e21-bdcd7a71fc25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.622430 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b579d6-2a88-4c63-9e21-bdcd7a71fc25 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.622712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b579d6-2a88-4c63-9e21-bdcd7a71fc25 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.622985 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b579d6-2a88-4c63-9e21-bdcd7a71fc25 admin admin] PATH_INFO: `/v3/projects/457dc83fb2094400a98bccf64a94b84e/users/86198a82904244dfa8eca9fcdc06b78f/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.638164 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b579d6-2a88-4c63-9e21-bdcd7a71fc25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=457dc83fb2094400a98bccf64a94b84e, user_id=86198a82904244dfa8eca9fcdc06b78f, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.640860 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b579d6-2a88-4c63-9e21-bdcd7a71fc25 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.654589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1283/2537] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:03:25 2022] PUT /identity/v3/projects/457dc83fb2094400a98bccf64a94b84e/users/86198a82904244dfa8eca9fcdc06b78f/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:25.658594 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345c94e6-e972-408b-b032-c97366c709ac admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.669225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345c94e6-e972-408b-b032-c97366c709ac admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.706021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:25.735048 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf1bec7-e968-4aca-80c8-54cc9b7f735f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.735772 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf1bec7-e968-4aca-80c8-54cc9b7f735f admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.736061 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf1bec7-e968-4aca-80c8-54cc9b7f735f admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.736381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf1bec7-e968-4aca-80c8-54cc9b7f735f admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.737021 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf1bec7-e968-4aca-80c8-54cc9b7f735f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.738770 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf1bec7-e968-4aca-80c8-54cc9b7f735f admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.750521 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1255/2538] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:25 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:03:25.754612 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b579d6-2a88-4c63-9e21-bdcd7a71fc25 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.765013 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b579d6-2a88-4c63-9e21-bdcd7a71fc25 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.769668 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.770327 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.770609 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.770886 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] PATH_INFO: `/v3/projects/457dc83fb2094400a98bccf64a94b84e/users/86198a82904244dfa8eca9fcdc06b78f/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.772848 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=457dc83fb2094400a98bccf64a94b84e, user_id=86198a82904244dfa8eca9fcdc06b78f, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:25.775482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:25.786450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1284/2539] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:03:25 2022] PUT /identity/v3/projects/457dc83fb2094400a98bccf64a94b84e/users/86198a82904244dfa8eca9fcdc06b78f/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:03:25.798585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf6a73-bf0e-4d31-88d9-b0c343f0e169 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.798807 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf6a73-bf0e-4d31-88d9-b0c343f0e169 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.799082 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf6a73-bf0e-4d31-88d9-b0c343f0e169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:25.827660 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-2ecf6a73-bf0e-4d31-88d9-b0c343f0e169 None None] MFA Rules not processed for user `86198a82904244dfa8eca9fcdc06b78f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:25.828950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-2ecf6a73-bf0e-4d31-88d9-b0c343f0e169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:25.899798 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1256/2540] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:25 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:25.910961 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:25.912792 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:25.985139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b87dad-3f0f-4625-a4ce-f103eac8c3a4 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:25.989181 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd4c42d-ad9c-44f3-9522-224070758658 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:25.989786 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd4c42d-ad9c-44f3-9522-224070758658 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:25.990063 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd4c42d-ad9c-44f3-9522-224070758658 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:25.990334 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd4c42d-ad9c-44f3-9522-224070758658 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:26.001809 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd4c42d-ad9c-44f3-9522-224070758658 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:26.003220 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd4c42d-ad9c-44f3-9522-224070758658 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:26.013358 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1285/2541] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:25 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:26.151840 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecf6a73-bf0e-4d31-88d9-b0c343f0e169 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:26.246963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecf6a73-bf0e-4d31-88d9-b0c343f0e169 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:26.251986 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:26.252669 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:26.252950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:26.253230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] PATH_INFO: `/v3/users/8497fed43a7746d28e288c64d18dc4a6` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:26.256275 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8497fed43a7746d28e288c64d18dc4a6)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:26.257456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:26.333357 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8497fed43a7746d28e288c64d18dc4a6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:26.342415 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8497fed43a7746d28e288c64d18dc4a6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:26.343709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:26.362027 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8497fed43a7746d28e288c64d18dc4a6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:26.362887 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8497fed43a7746d28e288c64d18dc4a6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:26.363596 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8497fed43a7746d28e288c64d18dc4a6'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:26.381775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1257/2542] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:03:26 2022] DELETE /identity/v3/users/8497fed43a7746d28e288c64d18dc4a6 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:26.386104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03defb86-4af8-4b2c-ad4f-0bf1bc869527 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:26.386594 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03defb86-4af8-4b2c-ad4f-0bf1bc869527 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:26.387310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03defb86-4af8-4b2c-ad4f-0bf1bc869527 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:26.406444 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-03defb86-4af8-4b2c-ad4f-0bf1bc869527 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:26.407648 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-03defb86-4af8-4b2c-ad4f-0bf1bc869527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:26.482482 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1286/2543] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:26.492622 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:26.494869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:26.555214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a81474a-5173-434c-8e1e-d68bf1de0fc8 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:26.559499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21698b-e503-4e22-a654-5083678c1765 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:26.560102 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21698b-e503-4e22-a654-5083678c1765 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:26.560424 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21698b-e503-4e22-a654-5083678c1765 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:26.560699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21698b-e503-4e22-a654-5083678c1765 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:26.569848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21698b-e503-4e22-a654-5083678c1765 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:26.571282 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21698b-e503-4e22-a654-5083678c1765 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:26.581913 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1258/2544] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 24 09:03:27.033004 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03defb86-4af8-4b2c-ad4f-0bf1bc869527 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:27.034652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-03defb86-4af8-4b2c-ad4f-0bf1bc869527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:27.048061 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03defb86-4af8-4b2c-ad4f-0bf1bc869527 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:27.052865 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:27.053577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:27.053922 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:27.054266 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] PATH_INFO: `/v3/projects/5226b786e04e4b6985bed73fa3322e6e` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:27.055389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5226b786e04e4b6985bed73fa3322e6e)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:27.057335 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:27.112766 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5226b786e04e4b6985bed73fa3322e6e', 'request_id': 'req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.119576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5226b786e04e4b6985bed73fa3322e6e', 'request_id': 'req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.128405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5226b786e04e4b6985bed73fa3322e6e', 'request_id': 'req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.130505 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:27.142317 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] The token cache is being invalidate because project 5226b786e04e4b6985bed73fa3322e6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:03:27.142620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.143682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.145634 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1287/2545] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:03:27 2022] DELETE /identity/v3/projects/5226b786e04e4b6985bed73fa3322e6e => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:27.150110 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21698b-e503-4e22-a654-5083678c1765 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:27.160548 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21698b-e503-4e22-a654-5083678c1765 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:27.222709 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:27.223444 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:27.223806 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:27.224171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] PATH_INFO: `/v3/users/86198a82904244dfa8eca9fcdc06b78f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:27.225780 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=86198a82904244dfa8eca9fcdc06b78f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:27.227408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:27.283511 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86198a82904244dfa8eca9fcdc06b78f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.293255 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86198a82904244dfa8eca9fcdc06b78f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.308440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86198a82904244dfa8eca9fcdc06b78f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.309900 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86198a82904244dfa8eca9fcdc06b78f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.310517 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-17bbda9c-5bca-4377-8b16-20a6968042c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86198a82904244dfa8eca9fcdc06b78f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:27.314819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:27.326527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1259/2546] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:03:27 2022] DELETE /identity/v3/users/86198a82904244dfa8eca9fcdc06b78f => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:27.895153 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:27.896660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:27.910489 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be3e1c4-bbca-4d3b-9bcb-fe4f617ad93d admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:27.984937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:27.985691 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:27.986038 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:27.986381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] PATH_INFO: `/v3/projects/457dc83fb2094400a98bccf64a94b84e` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:27.987733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=457dc83fb2094400a98bccf64a94b84e)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:27.989929 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:28.041422 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '457dc83fb2094400a98bccf64a94b84e', 'request_id': 'req-968dff6e-648c-4ed8-a814-b06747de5ba7'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:28.049841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '457dc83fb2094400a98bccf64a94b84e', 'request_id': 'req-968dff6e-648c-4ed8-a814-b06747de5ba7'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:28.060159 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '457dc83fb2094400a98bccf64a94b84e', 'request_id': 'req-968dff6e-648c-4ed8-a814-b06747de5ba7'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:28.075173 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] The token cache is being invalidate because project 457dc83fb2094400a98bccf64a94b84e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:03:28.075726 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:28.076652 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-968dff6e-648c-4ed8-a814-b06747de5ba7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:28.079240 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1288/2547] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:03:27 2022] DELETE /identity/v3/projects/457dc83fb2094400a98bccf64a94b84e => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:28.084075 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216414f-3ea9-4bd0-bce0-114cb670fdb6 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.084615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216414f-3ea9-4bd0-bce0-114cb670fdb6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.084735 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216414f-3ea9-4bd0-bce0-114cb670fdb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.107079 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e216414f-3ea9-4bd0-bce0-114cb670fdb6 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:28.108471 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e216414f-3ea9-4bd0-bce0-114cb670fdb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:28.190576 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1260/2548] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:28.198230 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b9732-bce1-4bed-bd77-9e90755c4361 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.198520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b9732-bce1-4bed-bd77-9e90755c4361 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.198793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b9732-bce1-4bed-bd77-9e90755c4361 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.222850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-a03b9732-bce1-4bed-bd77-9e90755c4361 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:28.225413 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-a03b9732-bce1-4bed-bd77-9e90755c4361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:28.252046 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1289/2549] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:28.256646 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e216414f-3ea9-4bd0-bce0-114cb670fdb6 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:28.273237 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e216414f-3ea9-4bd0-bce0-114cb670fdb6 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:28.281759 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:28.282486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.282852 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.283133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.283853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:03:28.284393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:28.288551 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:28.306236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1261/2550] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:03:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:03:28.310700 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03b9732-bce1-4bed-bd77-9e90755c4361 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:28.324048 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03b9732-bce1-4bed-bd77-9e90755c4361 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:28.330221 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065375d7-2e02-48bf-ac68-1ab9118c872c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:28.331112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065375d7-2e02-48bf-ac68-1ab9118c872c admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.331575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065375d7-2e02-48bf-ac68-1ab9118c872c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.332024 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065375d7-2e02-48bf-ac68-1ab9118c872c admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.332872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065375d7-2e02-48bf-ac68-1ab9118c872c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:28.335202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065375d7-2e02-48bf-ac68-1ab9118c872c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:28.366992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1290/2551] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:03:28 2022] POST /identity/v3/projects => generated 377 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:28.371854 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:28.383228 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c3102-34c5-4119-a5aa-86f295394e69 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:28.450455 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7c6f97-c46c-4c6a-973d-bc32b2b706d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:28.451404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c6f97-c46c-4c6a-973d-bc32b2b706d5 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.451689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c6f97-c46c-4c6a-973d-bc32b2b706d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.451961 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c6f97-c46c-4c6a-973d-bc32b2b706d5 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.452640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7c6f97-c46c-4c6a-973d-bc32b2b706d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:28.454758 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7c6f97-c46c-4c6a-973d-bc32b2b706d5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:28.483537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1262/2552] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:03:28 2022] POST /identity/v3/users => generated 289 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:28.484986 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065375d7-2e02-48bf-ac68-1ab9118c872c admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:28.495584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065375d7-2e02-48bf-ac68-1ab9118c872c admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:28.500323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b6e470-43b8-4002-af9e-3520caef43ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:28.501033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6e470-43b8-4002-af9e-3520caef43ff admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.501321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6e470-43b8-4002-af9e-3520caef43ff admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.501598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6e470-43b8-4002-af9e-3520caef43ff admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.502300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b6e470-43b8-4002-af9e-3520caef43ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:28.503980 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b6e470-43b8-4002-af9e-3520caef43ff admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:28.518435 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1291/2553] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:03:28.519911 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7c6f97-c46c-4c6a-973d-bc32b2b706d5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:28.530241 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7c6f97-c46c-4c6a-973d-bc32b2b706d5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:28.536656 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9350616-8e1f-4b37-a167-02b58b7567c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:28.537269 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9350616-8e1f-4b37-a167-02b58b7567c4 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.537549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9350616-8e1f-4b37-a167-02b58b7567c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.537821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9350616-8e1f-4b37-a167-02b58b7567c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.538468 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9350616-8e1f-4b37-a167-02b58b7567c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:28.540118 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9350616-8e1f-4b37-a167-02b58b7567c4 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:28.553372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1263/2554] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:03:28.557250 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b6e470-43b8-4002-af9e-3520caef43ff admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:28.569928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b6e470-43b8-4002-af9e-3520caef43ff admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:28.584790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ec90e3-cddd-4a46-953b-650f46dc9ff2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:28.585498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ec90e3-cddd-4a46-953b-650f46dc9ff2 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.585882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ec90e3-cddd-4a46-953b-650f46dc9ff2 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.586299 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ec90e3-cddd-4a46-953b-650f46dc9ff2 admin admin] PATH_INFO: `/v3/projects/12dfdb5034334587ac65060b041601b4/users/045af40adeda47e99a99da7b0ede259f/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.606097 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ec90e3-cddd-4a46-953b-650f46dc9ff2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=12dfdb5034334587ac65060b041601b4, user_id=045af40adeda47e99a99da7b0ede259f, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:28.609253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ec90e3-cddd-4a46-953b-650f46dc9ff2 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:28.624552 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1292/2555] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:03:28 2022] PUT /identity/v3/projects/12dfdb5034334587ac65060b041601b4/users/045af40adeda47e99a99da7b0ede259f/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:28.636355 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395e801f-480e-4e1c-8c53-7acace6f5862 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.637225 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395e801f-480e-4e1c-8c53-7acace6f5862 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.637627 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395e801f-480e-4e1c-8c53-7acace6f5862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.668994 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-395e801f-480e-4e1c-8c53-7acace6f5862 None None] MFA Rules not processed for user `045af40adeda47e99a99da7b0ede259f`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:28.670172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-395e801f-480e-4e1c-8c53-7acace6f5862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:28.718308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d4d8af-21bf-425d-8721-8dab06b27e74 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.718308 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d4d8af-21bf-425d-8721-8dab06b27e74 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.718663 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d4d8af-21bf-425d-8721-8dab06b27e74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.732677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:28.740964 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-73d4d8af-21bf-425d-8721-8dab06b27e74 None None] MFA Rules not processed for user `f452f27fc0d34f73aa22da2a09d54eb8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:28.742944 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-73d4d8af-21bf-425d-8721-8dab06b27e74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:28.757095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=89,keystone:DELETE=8,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:28.768937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1264/2556] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:28.773499 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395e801f-480e-4e1c-8c53-7acace6f5862 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:28.812087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1293/2557] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:28.868437 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395e801f-480e-4e1c-8c53-7acace6f5862 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:28.874331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbebdcf8-2a14-41a7-b16b-b79f2574a173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:28.875013 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbebdcf8-2a14-41a7-b16b-b79f2574a173 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.875380 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbebdcf8-2a14-41a7-b16b-b79f2574a173 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.875725 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbebdcf8-2a14-41a7-b16b-b79f2574a173 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.876457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbebdcf8-2a14-41a7-b16b-b79f2574a173 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:28.878793 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbebdcf8-2a14-41a7-b16b-b79f2574a173 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:28.913405 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1265/2558] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:03:28 2022] POST /identity/v3/projects => generated 377 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:28.916300 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d4d8af-21bf-425d-8721-8dab06b27e74 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:28.927930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d4d8af-21bf-425d-8721-8dab06b27e74 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:28.998129 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5a0e13-b37f-4d57-b65e-b39f8a501acb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:28.998778 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5a0e13-b37f-4d57-b65e-b39f8a501acb admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:28.999058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5a0e13-b37f-4d57-b65e-b39f8a501acb admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:28.999323 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5a0e13-b37f-4d57-b65e-b39f8a501acb admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:28.999989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5a0e13-b37f-4d57-b65e-b39f8a501acb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.001971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5a0e13-b37f-4d57-b65e-b39f8a501acb admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.027112 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1294/2559] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:03:28 2022] POST /identity/v3/users => generated 289 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:29.031601 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbebdcf8-2a14-41a7-b16b-b79f2574a173 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.043498 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbebdcf8-2a14-41a7-b16b-b79f2574a173 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.048266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb16ee7c-824d-48ff-86ec-5889398b8ffe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.048878 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16ee7c-824d-48ff-86ec-5889398b8ffe admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.049214 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16ee7c-824d-48ff-86ec-5889398b8ffe admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.049495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16ee7c-824d-48ff-86ec-5889398b8ffe admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.050342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb16ee7c-824d-48ff-86ec-5889398b8ffe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.052176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb16ee7c-824d-48ff-86ec-5889398b8ffe admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.063400 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1266/2560] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:03:29.067714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5a0e13-b37f-4d57-b65e-b39f8a501acb admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.078840 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5a0e13-b37f-4d57-b65e-b39f8a501acb admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.084149 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7157f9-9df6-4890-b2ff-8df53037faa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.084828 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7157f9-9df6-4890-b2ff-8df53037faa9 admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.085199 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7157f9-9df6-4890-b2ff-8df53037faa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.085534 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7157f9-9df6-4890-b2ff-8df53037faa9 admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.086249 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7157f9-9df6-4890-b2ff-8df53037faa9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.087928 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7157f9-9df6-4890-b2ff-8df53037faa9 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.099072 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1295/2561] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 24 09:03:29.103347 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb16ee7c-824d-48ff-86ec-5889398b8ffe admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.117706 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb16ee7c-824d-48ff-86ec-5889398b8ffe admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.126005 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e72146-323c-440e-aed4-a2d6207402f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.126700 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e72146-323c-440e-aed4-a2d6207402f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.127146 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e72146-323c-440e-aed4-a2d6207402f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.127674 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e72146-323c-440e-aed4-a2d6207402f5 admin admin] PATH_INFO: `/v3/projects/000c114766a1473cb8a1bb6adcd2234c/users/fc29a704baf848a58cb41e7bb659653f/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.152076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e72146-323c-440e-aed4-a2d6207402f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=000c114766a1473cb8a1bb6adcd2234c, user_id=fc29a704baf848a58cb41e7bb659653f, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.157105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e72146-323c-440e-aed4-a2d6207402f5 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.186420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1267/2562] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:03:29 2022] PUT /identity/v3/projects/000c114766a1473cb8a1bb6adcd2234c/users/fc29a704baf848a58cb41e7bb659653f/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:29.196847 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5949e0-c9e9-4e2a-adc8-640062fae246 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.197238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5949e0-c9e9-4e2a-adc8-640062fae246 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.197714 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5949e0-c9e9-4e2a-adc8-640062fae246 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.235703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-3c5949e0-c9e9-4e2a-adc8-640062fae246 None None] MFA Rules not processed for user `fc29a704baf848a58cb41e7bb659653f`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:29.237122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3c5949e0-c9e9-4e2a-adc8-640062fae246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:29.311138 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1296/2563] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:29.315848 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e72146-323c-440e-aed4-a2d6207402f5 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.385924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e72146-323c-440e-aed4-a2d6207402f5 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.391233 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f6d638-457e-4402-8515-638289c7ef40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.391864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f6d638-457e-4402-8515-638289c7ef40 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.392112 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f6d638-457e-4402-8515-638289c7ef40 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.392408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f6d638-457e-4402-8515-638289c7ef40 admin admin] PATH_INFO: `/v3/projects` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.393033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f6d638-457e-4402-8515-638289c7ef40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.395869 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f6d638-457e-4402-8515-638289c7ef40 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.448408 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1268/2564] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:03:29 2022] POST /identity/v3/projects => generated 375 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:29.453255 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5949e0-c9e9-4e2a-adc8-640062fae246 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.466169 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5949e0-c9e9-4e2a-adc8-640062fae246 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.508187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:29.540645 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06e2a71-8b6b-4fa5-aea3-51fa0ba07a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.541337 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06e2a71-8b6b-4fa5-aea3-51fa0ba07a6f admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.541683 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06e2a71-8b6b-4fa5-aea3-51fa0ba07a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.542051 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06e2a71-8b6b-4fa5-aea3-51fa0ba07a6f admin admin] PATH_INFO: `/v3/users` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.542757 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06e2a71-8b6b-4fa5-aea3-51fa0ba07a6f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.545363 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06e2a71-8b6b-4fa5-aea3-51fa0ba07a6f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.572170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1297/2565] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:03:29 2022] POST /identity/v3/users => generated 294 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:03:29.573933 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f6d638-457e-4402-8515-638289c7ef40 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.585146 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f6d638-457e-4402-8515-638289c7ef40 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.590587 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ea05fe-e1f7-498c-a42f-348204cb9e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.591442 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ea05fe-e1f7-498c-a42f-348204cb9e01 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.591790 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ea05fe-e1f7-498c-a42f-348204cb9e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.592010 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ea05fe-e1f7-498c-a42f-348204cb9e01 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.592763 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ea05fe-e1f7-498c-a42f-348204cb9e01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.594369 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ea05fe-e1f7-498c-a42f-348204cb9e01 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.609485 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1269/2566] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:03:29.610643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06e2a71-8b6b-4fa5-aea3-51fa0ba07a6f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.620615 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06e2a71-8b6b-4fa5-aea3-51fa0ba07a6f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.626186 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9befb-5f7b-4c9c-b814-48ebcc4a40f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.626843 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9befb-5f7b-4c9c-b814-48ebcc4a40f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.627122 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9befb-5f7b-4c9c-b814-48ebcc4a40f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.627390 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9befb-5f7b-4c9c-b814-48ebcc4a40f3 admin admin] PATH_INFO: `/v3/projects/3e94a88821bf4999bb11bab688804a70/users/952790b51f8f46c2b6de6ce0b023490c/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.645682 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc9befb-5f7b-4c9c-b814-48ebcc4a40f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e94a88821bf4999bb11bab688804a70, user_id=952790b51f8f46c2b6de6ce0b023490c, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.648584 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc9befb-5f7b-4c9c-b814-48ebcc4a40f3 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.664532 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1298/2567] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:03:29 2022] PUT /identity/v3/projects/3e94a88821bf4999bb11bab688804a70/users/952790b51f8f46c2b6de6ce0b023490c/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:29.669148 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ea05fe-e1f7-498c-a42f-348204cb9e01 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.686497 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ea05fe-e1f7-498c-a42f-348204cb9e01 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.699076 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:29.755147 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd4d3ef-8576-4a34-a508-559f92cdb128 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.756101 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd4d3ef-8576-4a34-a508-559f92cdb128 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.756271 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd4d3ef-8576-4a34-a508-559f92cdb128 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.756761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd4d3ef-8576-4a34-a508-559f92cdb128 admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.757316 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd4d3ef-8576-4a34-a508-559f92cdb128 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.759372 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd4d3ef-8576-4a34-a508-559f92cdb128 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.770272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1270/2568] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:03:29.774373 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9befb-5f7b-4c9c-b814-48ebcc4a40f3 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.785583 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9befb-5f7b-4c9c-b814-48ebcc4a40f3 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:29.790339 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:29.790988 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.791321 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.791657 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] PATH_INFO: `/v3/projects/3e94a88821bf4999bb11bab688804a70/users/952790b51f8f46c2b6de6ce0b023490c/roles/73b89f24867f4f45937bb4a67856e0ba` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.794238 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e94a88821bf4999bb11bab688804a70, user_id=952790b51f8f46c2b6de6ce0b023490c, role_id=73b89f24867f4f45937bb4a67856e0ba)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:29.797354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:29.809437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1299/2569] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:03:29 2022] PUT /identity/v3/projects/3e94a88821bf4999bb11bab688804a70/users/952790b51f8f46c2b6de6ce0b023490c/roles/73b89f24867f4f45937bb4a67856e0ba => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:29.819560 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab2846-9dfb-44ff-8b8d-a0b6ed574262 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:29.819850 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab2846-9dfb-44ff-8b8d-a0b6ed574262 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:29.820160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab2846-9dfb-44ff-8b8d-a0b6ed574262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:29.854581 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-e1ab2846-9dfb-44ff-8b8d-a0b6ed574262 None None] MFA Rules not processed for user `952790b51f8f46c2b6de6ce0b023490c`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:29.856204 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e1ab2846-9dfb-44ff-8b8d-a0b6ed574262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:29.928251 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1271/2570] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:29 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:29.939535 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:29.941294 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:30.002131 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22ddc3-a2e8-45cc-918f-fe143b903a03 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:30.006310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec4b81-3797-4c95-9768-6f4a8e18de7c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:30.006921 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec4b81-3797-4c95-9768-6f4a8e18de7c service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:30.007204 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec4b81-3797-4c95-9768-6f4a8e18de7c service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:30.007477 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec4b81-3797-4c95-9768-6f4a8e18de7c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:30.064211 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec4b81-3797-4c95-9768-6f4a8e18de7c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:30.065576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec4b81-3797-4c95-9768-6f4a8e18de7c service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:30.074813 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1300/2571] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:31.588479 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ab2846-9dfb-44ff-8b8d-a0b6ed574262 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:31.590552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e1ab2846-9dfb-44ff-8b8d-a0b6ed574262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:31.602809 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ab2846-9dfb-44ff-8b8d-a0b6ed574262 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:31.607163 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e833ec-73bc-4a94-8187-c39fce4cee3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:31.607832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e833ec-73bc-4a94-8187-c39fce4cee3b service cinder] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:31.608211 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e833ec-73bc-4a94-8187-c39fce4cee3b service cinder] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:31.608578 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e833ec-73bc-4a94-8187-c39fce4cee3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:31.674396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e833ec-73bc-4a94-8187-c39fce4cee3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:31.675766 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e833ec-73bc-4a94-8187-c39fce4cee3b service cinder] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:31.685139 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1272/2572] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:31 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:32.267206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec4b81-3797-4c95-9768-6f4a8e18de7c service cinder] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:32.276085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec4b81-3797-4c95-9768-6f4a8e18de7c service cinder] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:32.280316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652d5770-c8db-4e62-b99c-c326a9885424 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:32.280919 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d5770-c8db-4e62-b99c-c326a9885424 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:32.281197 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d5770-c8db-4e62-b99c-c326a9885424 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:32.281470 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d5770-c8db-4e62-b99c-c326a9885424 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:32.290814 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d5770-c8db-4e62-b99c-c326a9885424 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:32.292362 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d5770-c8db-4e62-b99c-c326a9885424 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:32.302712 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1301/2573] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:32 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:36.197424 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e833ec-73bc-4a94-8187-c39fce4cee3b service cinder] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:36.236526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=100 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:36.260821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e833ec-73bc-4a94-8187-c39fce4cee3b service cinder] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:36.265877 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:36.266495 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:36.266776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:36.267077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] PATH_INFO: `/v3/users/045af40adeda47e99a99da7b0ede259f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:36.268209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] RBAC: Authorizing `identity:delete_user(user_id=045af40adeda47e99a99da7b0ede259f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:36.270124 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:36.316480 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '045af40adeda47e99a99da7b0ede259f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:36.323159 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '045af40adeda47e99a99da7b0ede259f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:36.335672 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '045af40adeda47e99a99da7b0ede259f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:36.336378 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '045af40adeda47e99a99da7b0ede259f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:36.336987 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '045af40adeda47e99a99da7b0ede259f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:36.351661 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1273/2574] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:03:36 2022] DELETE /identity/v3/users/045af40adeda47e99a99da7b0ede259f => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:36.356068 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6839ed14-ae75-403a-bce3-836df413e2dd None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:36.356564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6839ed14-ae75-403a-bce3-836df413e2dd None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:36.356780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6839ed14-ae75-403a-bce3-836df413e2dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:36.376877 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-6839ed14-ae75-403a-bce3-836df413e2dd None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:36.378114 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6839ed14-ae75-403a-bce3-836df413e2dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:36.395878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:36.440311 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1302/2575] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:03:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:03:36.449819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:36.451261 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:36.509966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacd349c-3894-41a0-b93f-997887b2d053 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:36.514190 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57cf663-a286-40d0-ad5b-cfc16cf13b01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:36.514792 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57cf663-a286-40d0-ad5b-cfc16cf13b01 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:36.515074 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57cf663-a286-40d0-ad5b-cfc16cf13b01 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:36.515350 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57cf663-a286-40d0-ad5b-cfc16cf13b01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:36.524610 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57cf663-a286-40d0-ad5b-cfc16cf13b01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:36.526037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57cf663-a286-40d0-ad5b-cfc16cf13b01 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:36.536940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1274/2576] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:03:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:03:36.983998 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6839ed14-ae75-403a-bce3-836df413e2dd None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:36.985936 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-6839ed14-ae75-403a-bce3-836df413e2dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:37.006727 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6839ed14-ae75-403a-bce3-836df413e2dd None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:37.011833 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:37.013283 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:37.013871 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:37.014213 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] PATH_INFO: `/v3/projects/12dfdb5034334587ac65060b041601b4` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:37.021108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] RBAC: Authorizing `identity:delete_project(project_id=12dfdb5034334587ac65060b041601b4)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:37.023052 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:37.073821 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '12dfdb5034334587ac65060b041601b4', 'request_id': 'req-681c24ef-ef52-4a61-8383-5ed0a2a27463'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.081270 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '12dfdb5034334587ac65060b041601b4', 'request_id': 'req-681c24ef-ef52-4a61-8383-5ed0a2a27463'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.089849 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '12dfdb5034334587ac65060b041601b4', 'request_id': 'req-681c24ef-ef52-4a61-8383-5ed0a2a27463'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.100620 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] The token cache is being invalidate because project 12dfdb5034334587ac65060b041601b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:03:37.100911 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.101538 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.103226 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1303/2577] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:03:36 2022] DELETE /identity/v3/projects/12dfdb5034334587ac65060b041601b4 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:37.107585 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57cf663-a286-40d0-ad5b-cfc16cf13b01 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:37.117666 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57cf663-a286-40d0-ad5b-cfc16cf13b01 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:37.175947 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:37.182813 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:37.184040 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:37.184483 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:37.184821 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] PATH_INFO: `/v3/users/fc29a704baf848a58cb41e7bb659653f` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:37.186331 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc29a704baf848a58cb41e7bb659653f)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:37.188936 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:37.234864 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc29a704baf848a58cb41e7bb659653f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.242093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc29a704baf848a58cb41e7bb659653f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.254365 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc29a704baf848a58cb41e7bb659653f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.255057 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc29a704baf848a58cb41e7bb659653f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.255655 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc29a704baf848a58cb41e7bb659653f'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.268661 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1275/2578] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:03:37 2022] DELETE /identity/v3/users/fc29a704baf848a58cb41e7bb659653f => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:37.539331 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:37.541333 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:37.562426 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c24ef-ef52-4a61-8383-5ed0a2a27463 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:37.617278 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:37.617898 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:37.618177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:37.618450 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] PATH_INFO: `/v3/projects/000c114766a1473cb8a1bb6adcd2234c` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:37.619859 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=000c114766a1473cb8a1bb6adcd2234c)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:37.622485 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:37.670520 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '000c114766a1473cb8a1bb6adcd2234c', 'request_id': 'req-59f0f30f-7491-40f0-bf99-d4440759fd05'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.671529 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:37.678492 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '000c114766a1473cb8a1bb6adcd2234c', 'request_id': 'req-59f0f30f-7491-40f0-bf99-d4440759fd05'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.687058 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '000c114766a1473cb8a1bb6adcd2234c', 'request_id': 'req-59f0f30f-7491-40f0-bf99-d4440759fd05'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.698202 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] The token cache is being invalidate because project 000c114766a1473cb8a1bb6adcd2234c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:03:37.698498 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.699139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.700772 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1304/2579] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:03:37 2022] DELETE /identity/v3/projects/000c114766a1473cb8a1bb6adcd2234c => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:37.704905 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:37.706640 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:37.720037 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576f468-8e73-45d6-b2f3-c48e605ef4c1 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:37.780909 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:37.781543 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:37.781819 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:37.782089 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] PATH_INFO: `/v3/users/952790b51f8f46c2b6de6ce0b023490c` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:37.783288 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=952790b51f8f46c2b6de6ce0b023490c)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:37.785342 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:37.831777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '952790b51f8f46c2b6de6ce0b023490c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.838670 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '952790b51f8f46c2b6de6ce0b023490c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.850937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '952790b51f8f46c2b6de6ce0b023490c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.851590 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '952790b51f8f46c2b6de6ce0b023490c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.852209 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-3554f22e-b8bd-4d5d-96fa-5c33c2352dd3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '952790b51f8f46c2b6de6ce0b023490c'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:37.856544 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:37.867984 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1276/2580] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:03:37 2022] DELETE /identity/v3/users/952790b51f8f46c2b6de6ce0b023490c => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:38.135402 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:38.137467 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:38.153375 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f0f30f-7491-40f0-bf99-d4440759fd05 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:38.212949 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:38.213589 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:38.213868 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:38.214136 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] PATH_INFO: `/v3/projects/3e94a88821bf4999bb11bab688804a70` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:38.216431 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e94a88821bf4999bb11bab688804a70)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:38.218313 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:38.269005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e94a88821bf4999bb11bab688804a70', 'request_id': 'req-c52625ec-a3f7-4a4f-9a73-615034181908'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:38.275355 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e94a88821bf4999bb11bab688804a70', 'request_id': 'req-c52625ec-a3f7-4a4f-9a73-615034181908'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:38.285040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e94a88821bf4999bb11bab688804a70', 'request_id': 'req-c52625ec-a3f7-4a4f-9a73-615034181908'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:38.296522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] The token cache is being invalidate because project 3e94a88821bf4999bb11bab688804a70 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:03:38.296869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:38.297527 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:03:38.299292 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1305/2581] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:03:38 2022] DELETE /identity/v3/projects/3e94a88821bf4999bb11bab688804a70 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:03:47.866294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=2 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:48.292753 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=57,keystone:INSERT=2,keystone:DELETE=10 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:03:56.284880 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4b26d-d6f7-4d8f-ad18-0af637fa2be6 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:56.285340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4b26d-d6f7-4d8f-ad18-0af637fa2be6 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:56.285535 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4b26d-d6f7-4d8f-ad18-0af637fa2be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:56.305344 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-80a4b26d-d6f7-4d8f-ad18-0af637fa2be6 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:03:56.306532 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-80a4b26d-d6f7-4d8f-ad18-0af637fa2be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:56.361956 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1277/2582] 10.222.0.39 () {58 vars in 1009 bytes} [Mon Oct 24 09:03:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:03:56.402703 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:03:56.404200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:03:56.474725 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52625ec-a3f7-4a4f-9a73-615034181908 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:03:56.479874 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4712a300-fa90-4dcf-bd87-b344b57c8faa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:03:56.480516 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4712a300-fa90-4dcf-bd87-b344b57c8faa service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:03:56.481448 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4712a300-fa90-4dcf-bd87-b344b57c8faa service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:03:56.481728 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4712a300-fa90-4dcf-bd87-b344b57c8faa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:03:56.489937 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4712a300-fa90-4dcf-bd87-b344b57c8faa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:03:56.491512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4712a300-fa90-4dcf-bd87-b344b57c8faa service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:03:56.500481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1306/2583] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 09:03:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 09:04:06.355196 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=25 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:04:06.497042 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=32 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:04:11.737891 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e718b42-ac8e-4b9c-87a5-41c90ddbabe5 None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:11.738456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e718b42-ac8e-4b9c-87a5-41c90ddbabe5 None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:11.738456 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e718b42-ac8e-4b9c-87a5-41c90ddbabe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:11.758093 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7e718b42-ac8e-4b9c-87a5-41c90ddbabe5 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:04:11.759313 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7e718b42-ac8e-4b9c-87a5-41c90ddbabe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:11.766990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1278/2584] 10.222.0.39 () {58 vars in 967 bytes} [Mon Oct 24 09:04:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:04:11.779340 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4712a300-fa90-4dcf-bd87-b344b57c8faa service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:11.789824 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4712a300-fa90-4dcf-bd87-b344b57c8faa service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:11.795253 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707d7927-e22f-4143-8738-ae8c772e6bc5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:11.795830 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d7927-e22f-4143-8738-ae8c772e6bc5 service swift] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:11.796104 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d7927-e22f-4143-8738-ae8c772e6bc5 service swift] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:11.796427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d7927-e22f-4143-8738-ae8c772e6bc5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:11.804116 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707d7927-e22f-4143-8738-ae8c772e6bc5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:11.805692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707d7927-e22f-4143-8738-ae8c772e6bc5 service swift] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:11.814395 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1307/2585] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 09:04:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 09:04:13.731657 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e718b42-ac8e-4b9c-87a5-41c90ddbabe5 None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:13.733401 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7e718b42-ac8e-4b9c-87a5-41c90ddbabe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:13.803232 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e718b42-ac8e-4b9c-87a5-41c90ddbabe5 None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:13.807740 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:13.808393 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:13.808677 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:13.808950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] PATH_INFO: `/v3/users/f452f27fc0d34f73aa22da2a09d54eb8` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:13.810077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f452f27fc0d34f73aa22da2a09d54eb8)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:13.811975 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:13.858628 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f452f27fc0d34f73aa22da2a09d54eb8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:13.865476 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f452f27fc0d34f73aa22da2a09d54eb8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:13.877716 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f452f27fc0d34f73aa22da2a09d54eb8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:13.878382 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f452f27fc0d34f73aa22da2a09d54eb8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:13.879007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-fcb290bb-c9a6-4653-9928-a7d4ac70e5d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f452f27fc0d34f73aa22da2a09d54eb8'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:13.891501 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1279/2586] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:04:13 2022] DELETE /identity/v3/users/f452f27fc0d34f73aa22da2a09d54eb8 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:04:14.066781 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707d7927-e22f-4143-8738-ae8c772e6bc5 service swift] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:14.079576 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707d7927-e22f-4143-8738-ae8c772e6bc5 service swift] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:14.143187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:14.143807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.144086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.144389 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] PATH_INFO: `/v3/projects/d0ccb79814f5432cb21c619e974fecd8` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.145481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0ccb79814f5432cb21c619e974fecd8)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:14.147351 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:14.195037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0ccb79814f5432cb21c619e974fecd8', 'request_id': 'req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:14.201854 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0ccb79814f5432cb21c619e974fecd8', 'request_id': 'req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:14.209487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0ccb79814f5432cb21c619e974fecd8', 'request_id': 'req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:14.219689 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] The token cache is being invalidate because project d0ccb79814f5432cb21c619e974fecd8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:04:14.219974 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:14.220629 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-5d243d65-bfe3-4f10-b3f6-03b78fd0849c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:14.222085 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1308/2587] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:04:14 2022] DELETE /identity/v3/projects/d0ccb79814f5432cb21c619e974fecd8 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:04:14.226872 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da011deb-b3d1-4da9-9fae-7f0587ec50cc None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.227155 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da011deb-b3d1-4da9-9fae-7f0587ec50cc None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.227423 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da011deb-b3d1-4da9-9fae-7f0587ec50cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.247785 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-da011deb-b3d1-4da9-9fae-7f0587ec50cc None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:04:14.249006 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-da011deb-b3d1-4da9-9fae-7f0587ec50cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:14.306427 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1280/2588] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:04:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:04:14.315163 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671d4383-3215-459b-99b5-9c75cbadc98c None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.315443 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671d4383-3215-459b-99b5-9c75cbadc98c None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.315713 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671d4383-3215-459b-99b5-9c75cbadc98c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.335850 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-671d4383-3215-459b-99b5-9c75cbadc98c None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:04:14.337142 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-671d4383-3215-459b-99b5-9c75cbadc98c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:14.345958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1309/2589] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:04:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:04:14.350156 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da011deb-b3d1-4da9-9fae-7f0587ec50cc None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:14.359733 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da011deb-b3d1-4da9-9fae-7f0587ec50cc None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:14.364183 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:14.364774 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.365055 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.365324 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] PATH_INFO: `/v3/domains` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.365957 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=93079) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 24 09:04:14.366266 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:14.367871 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:14.376922 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:04:14.380721 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1281/2590] 10.222.0.39 () {58 vars in 1153 bytes} [Mon Oct 24 09:04:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 24 09:04:14.385089 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671d4383-3215-459b-99b5-9c75cbadc98c None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:14.394940 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671d4383-3215-459b-99b5-9c75cbadc98c None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:14.399564 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e9e2b-5867-4772-8925-47fc230450ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:14.400206 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e9e2b-5867-4772-8925-47fc230450ba admin admin] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.400490 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e9e2b-5867-4772-8925-47fc230450ba admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.400761 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e9e2b-5867-4772-8925-47fc230450ba admin admin] PATH_INFO: `/v3/projects` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.401388 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e9e2b-5867-4772-8925-47fc230450ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:14.403878 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e9e2b-5867-4772-8925-47fc230450ba admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:14.430447 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1310/2591] 10.222.0.39 () {60 vars in 1153 bytes} [Mon Oct 24 09:04:14 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 24 09:04:14.434592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:14.448409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493abce2-e478-4383-8a06-5f3263deb8c0 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:14.505459 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc8833-c825-4d59-9b00-1a5f4c741f01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:14.506059 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc8833-c825-4d59-9b00-1a5f4c741f01 admin admin] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.506340 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc8833-c825-4d59-9b00-1a5f4c741f01 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.506615 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc8833-c825-4d59-9b00-1a5f4c741f01 admin admin] PATH_INFO: `/v3/users` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.507247 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcc8833-c825-4d59-9b00-1a5f4c741f01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:14.509180 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcc8833-c825-4d59-9b00-1a5f4c741f01 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:14.530559 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1282/2592] 10.222.0.39 () {60 vars in 1144 bytes} [Mon Oct 24 09:04:14 2022] POST /identity/v3/users => generated 287 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 24 09:04:14.534674 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e9e2b-5867-4772-8925-47fc230450ba admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:14.544665 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e9e2b-5867-4772-8925-47fc230450ba admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:14.549225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d1a81d-eced-4efb-a71c-68437155eecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:14.549807 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d1a81d-eced-4efb-a71c-68437155eecc admin admin] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.550086 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d1a81d-eced-4efb-a71c-68437155eecc admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.550354 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d1a81d-eced-4efb-a71c-68437155eecc admin admin] PATH_INFO: `/v3/roles` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.550989 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d1a81d-eced-4efb-a71c-68437155eecc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:14.552600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d1a81d-eced-4efb-a71c-68437155eecc admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:14.562917 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1311/2593] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:04:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:04:14.566752 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc8833-c825-4d59-9b00-1a5f4c741f01 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:14.577643 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc8833-c825-4d59-9b00-1a5f4c741f01 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:14.582172 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6180635-5f63-40f3-a52d-82baa2002a7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:14.582747 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6180635-5f63-40f3-a52d-82baa2002a7d admin admin] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.583024 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6180635-5f63-40f3-a52d-82baa2002a7d admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.583293 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6180635-5f63-40f3-a52d-82baa2002a7d admin admin] PATH_INFO: `/v3/roles` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.583914 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6180635-5f63-40f3-a52d-82baa2002a7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:14.585516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6180635-5f63-40f3-a52d-82baa2002a7d admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:14.596358 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1283/2594] 10.222.0.39 () {58 vars in 1122 bytes} [Mon Oct 24 09:04:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 24 09:04:14.600302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d1a81d-eced-4efb-a71c-68437155eecc admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:14.610366 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d1a81d-eced-4efb-a71c-68437155eecc admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:14.614962 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:14.615593 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] REQUEST_METHOD: `PUT` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.615872 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.616165 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] PATH_INFO: `/v3/projects/e895912921364884a6e4c39d1c094cd4/users/9808d79b77e94044a6ff8ed8c304b597/roles/05482458bad348e0be384b75bfd31eac` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.631200 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] RBAC: Authorizing `identity:create_grant(project_id=e895912921364884a6e4c39d1c094cd4, user_id=9808d79b77e94044a6ff8ed8c304b597, role_id=05482458bad348e0be384b75bfd31eac)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:14.633930 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:14.647005 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1312/2595] 10.222.0.39 () {60 vars in 1483 bytes} [Mon Oct 24 09:04:14 2022] PUT /identity/v3/projects/e895912921364884a6e4c39d1c094cd4/users/9808d79b77e94044a6ff8ed8c304b597/roles/05482458bad348e0be384b75bfd31eac => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:04:14.656974 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4fc3f-e2f9-4863-b011-6804cec58a7e None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.657268 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4fc3f-e2f9-4863-b011-6804cec58a7e None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.657546 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4fc3f-e2f9-4863-b011-6804cec58a7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.684873 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-7db4fc3f-e2f9-4863-b011-6804cec58a7e None None] MFA Rules not processed for user `9808d79b77e94044a6ff8ed8c304b597`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:04:14.686078 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-7db4fc3f-e2f9-4863-b011-6804cec58a7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:14.749624 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1284/2596] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:04:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:04:14.760177 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:14.761856 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:14.776558 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:04:14.820837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd10727-6f68-48f4-a076-c21beee5c1af admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:14.825307 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7128d3e1-4f7f-4441-b9ef-385755592445 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:14.825973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7128d3e1-4f7f-4441-b9ef-385755592445 service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:14.826279 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7128d3e1-4f7f-4441-b9ef-385755592445 service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:14.826551 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7128d3e1-4f7f-4441-b9ef-385755592445 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:14.835598 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7128d3e1-4f7f-4441-b9ef-385755592445 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:14.837040 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7128d3e1-4f7f-4441-b9ef-385755592445 service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:14.846219 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1313/2597] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:04:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:04:24.746270 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=66,keystone:INSERT=3 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:04:24.842035 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=20 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:04:24.909553 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db4fc3f-e2f9-4863-b011-6804cec58a7e None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:24.970537 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db4fc3f-e2f9-4863-b011-6804cec58a7e None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:24.975063 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:24.975679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:24.975958 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:24.976262 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] PATH_INFO: `/v3/users/9808d79b77e94044a6ff8ed8c304b597` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:24.977383 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=9808d79b77e94044a6ff8ed8c304b597)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:24.979294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:25.026926 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9808d79b77e94044a6ff8ed8c304b597'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.034165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9808d79b77e94044a6ff8ed8c304b597'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.045916 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9808d79b77e94044a6ff8ed8c304b597'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.046591 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9808d79b77e94044a6ff8ed8c304b597'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.047201 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9808d79b77e94044a6ff8ed8c304b597'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.060212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1285/2598] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:04:24 2022] DELETE /identity/v3/users/9808d79b77e94044a6ff8ed8c304b597 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:04:25.064733 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4a9a33-e0bc-4b45-a39f-b37eec613051 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:25.065074 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4a9a33-e0bc-4b45-a39f-b37eec613051 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:25.065359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4a9a33-e0bc-4b45-a39f-b37eec613051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:25.084642 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-0f4a9a33-e0bc-4b45-a39f-b37eec613051 None None] MFA Rules not processed for user `0bb4ebf18f65413bbe4945aec1caead8`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:04:25.085838 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0f4a9a33-e0bc-4b45-a39f-b37eec613051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:25.145491 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1314/2599] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:04:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 24 09:04:25.155592 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:25.157039 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:25.215552 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d76084e-81d1-4267-aaac-b0fb5ac3eb6a admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:25.219526 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:25.220160 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:25.220445 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:25.220722 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:25.230937 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:25.232990 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:25.243272 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1286/2600] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:04:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:04:25.599577 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4a9a33-e0bc-4b45-a39f-b37eec613051 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:04:25.601082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-0f4a9a33-e0bc-4b45-a39f-b37eec613051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:04:25.622571 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4a9a33-e0bc-4b45-a39f-b37eec613051 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:04:25.633500 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:04:25.634322 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:04:25.634600 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:04:25.634869 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] PATH_INFO: `/v3/projects/e895912921364884a6e4c39d1c094cd4` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:04:25.636392 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=e895912921364884a6e4c39d1c094cd4)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:04:25.639053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:04:25.686179 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e895912921364884a6e4c39d1c094cd4', 'request_id': 'req-c63a5eb6-93a2-48c8-9839-7f6a201389eb'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.692380 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e895912921364884a6e4c39d1c094cd4', 'request_id': 'req-c63a5eb6-93a2-48c8-9839-7f6a201389eb'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.700433 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e895912921364884a6e4c39d1c094cd4', 'request_id': 'req-c63a5eb6-93a2-48c8-9839-7f6a201389eb'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.711120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] The token cache is being invalidate because project e895912921364884a6e4c39d1c094cd4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:04:25.711437 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.712081 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:04:25.713654 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1315/2601] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:04:25 2022] DELETE /identity/v3/projects/e895912921364884a6e4c39d1c094cd4 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:04:35.239195 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:04:35.708095 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:05:04.288637 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:04.290122 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:04.358985 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a896d1b0-c632-4f5a-9107-a492679a1e5c service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:04.419339 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fd32f8-fb99-4036-a31a-116198e17d56 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:04.419924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd32f8-fb99-4036-a31a-116198e17d56 service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:04.420257 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd32f8-fb99-4036-a31a-116198e17d56 service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:04.420530 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd32f8-fb99-4036-a31a-116198e17d56 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:04.422230 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-81fd32f8-fb99-4036-a31a-116198e17d56 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:04.433106 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fd32f8-fb99-4036-a31a-116198e17d56 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:04.435317 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fd32f8-fb99-4036-a31a-116198e17d56 service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:04.445000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1287/2602] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:05:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:05:14.440762 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=58 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:05:35.244591 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:35.246043 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:35.302776 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a5eb6-93a2-48c8-9839-7f6a201389eb admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:35.306297 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8eb0495-e216-4962-be4f-bda9d06d27dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '412f520f90564e44819e8bf947992efb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:35.306892 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eb0495-e216-4962-be4f-bda9d06d27dd service cinder] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:35.307170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eb0495-e216-4962-be4f-bda9d06d27dd service cinder] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:35.307442 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eb0495-e216-4962-be4f-bda9d06d27dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:35.309171 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d8eb0495-e216-4962-be4f-bda9d06d27dd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:35.369302 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eb0495-e216-4962-be4f-bda9d06d27dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:35.370718 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eb0495-e216-4962-be4f-bda9d06d27dd service cinder] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:35.379890 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1316/2603] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:05:35 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:05:45.375748 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=48 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:05:45.431474 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f3aac-9218-432c-a099-be196457428c None None] REQUEST_METHOD: `POST` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:45.432017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f3aac-9218-432c-a099-be196457428c None None] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:45.432017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f3aac-9218-432c-a099-be196457428c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:45.453254 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.auth.core [None req-1d8f3aac-9218-432c-a099-be196457428c None None] MFA Rules not processed for user `3cddd27c37514e74a38430c46cef16fa`. Rule list: `[]` (Enabled: `True`). {{(pid=93079) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:05:45.454440 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1d8f3aac-9218-432c-a099-be196457428c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:45.461067 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1288/2604] 10.222.0.39 () {58 vars in 958 bytes} [Mon Oct 24 09:05:45 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:05:51.349273 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29dd5f2-22b5-459b-8260-02fc04e26812 None None] REQUEST_METHOD: `POST` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:51.349738 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29dd5f2-22b5-459b-8260-02fc04e26812 None None] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:51.349973 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29dd5f2-22b5-459b-8260-02fc04e26812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:51.369656 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.auth.core [None req-d29dd5f2-22b5-459b-8260-02fc04e26812 None None] MFA Rules not processed for user `128903595c5e454c8c51c371d014eb49`. Rule list: `[]` (Enabled: `True`). {{(pid=93080) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 24 09:05:51.370858 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d29dd5f2-22b5-459b-8260-02fc04e26812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:51.431606 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1317/2605] 10.222.0.39 () {58 vars in 967 bytes} [Mon Oct 24 09:05:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 24 09:05:51.482690 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8f3aac-9218-432c-a099-be196457428c None None] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:51.484154 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1d8f3aac-9218-432c-a099-be196457428c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:51.558502 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8f3aac-9218-432c-a099-be196457428c None None] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:51.564060 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60a07d37339f4f83ad2ce8be3d0892a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:51.564688 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:51.564969 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:51.565242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:51.573263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:51.575460 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:51.583617 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1289/2606] 10.222.0.39 () {60 vars in 1384 bytes} [Mon Oct 24 09:05:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 24 09:05:51.849037 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29dd5f2-22b5-459b-8260-02fc04e26812 None None] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:51.850504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-d29dd5f2-22b5-459b-8260-02fc04e26812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:51.909127 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29dd5f2-22b5-459b-8260-02fc04e26812 None None] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:51.913160 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:51.913915 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:51.914192 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:51.914461 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:51.916289 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:51.997397 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:51.999243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:52.010286 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1318/2607] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:05:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:05:55.318941 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:55.320398 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:55.336950 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e583e97d-46e5-41f0-ae95-46baa5d3a497 service swift] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:55.343614 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:55.344263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:55.344802 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:55.345077 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:55.346839 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:55.359976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:55.361714 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:55.374105 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1290/2608] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:05:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:05:55.837476 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:55.847087 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4571bbcd-57da-456a-953d-cabad4dd52b0 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:55.851621 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:55.852522 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:55.852958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:55.853427 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:55.855316 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:55.870420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:55.872956 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:55.885008 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1319/2609] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:05:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:05:58.351668 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:58.364531 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1152bd02-d29a-4ef0-bcf2-ef19877381c9 service placement] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:58.369924 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:58.370584 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:58.371012 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:58.371295 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:58.375979 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:58.386964 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:58.389496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:58.417243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1291/2610] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:05:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:05:58.432842 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:58.434595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:58.480225 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:05:58.513920 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc64e73-c3d4-4767-92ae-3380b909f35b service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:58.518514 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:58.519135 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:58.519416 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:58.519692 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] PATH_INFO: `/v3/users/4f104bf97764426cb67b14bbf39e16be` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:58.520811 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f104bf97764426cb67b14bbf39e16be)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:58.522719 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:58.567970 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f104bf97764426cb67b14bbf39e16be'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.574981 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f104bf97764426cb67b14bbf39e16be'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.587731 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f104bf97764426cb67b14bbf39e16be'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.588575 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f104bf97764426cb67b14bbf39e16be'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.589195 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f104bf97764426cb67b14bbf39e16be'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.602633 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1320/2611] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:05:58 2022] DELETE /identity/v3/users/4f104bf97764426cb67b14bbf39e16be => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:05:58.779216 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:58.795409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdb87ce-a29c-4856-b658-5c839482fc9d service placement] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:58.856853 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:58.857404 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:58.857689 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:58.857968 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] PATH_INFO: `/v3/projects/7e9c4dec37674f73912f0d9272ac2446` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:58.859165 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e9c4dec37674f73912f0d9272ac2446)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:58.861119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:58.907632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e9c4dec37674f73912f0d9272ac2446', 'request_id': 'req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.913133 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e9c4dec37674f73912f0d9272ac2446', 'request_id': 'req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.914192 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:05:58.921150 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e9c4dec37674f73912f0d9272ac2446', 'request_id': 'req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.933375 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] The token cache is being invalidate because project 7e9c4dec37674f73912f0d9272ac2446 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:05:58.933811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.935126 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:58.936556 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1292/2612] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:05:58 2022] DELETE /identity/v3/projects/7e9c4dec37674f73912f0d9272ac2446 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:05:58.940613 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:58.941992 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:58.958721 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce60292f-51fc-4ed3-b56e-097ec6b52730 admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:59.017208 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:59.017834 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:59.018121 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:59.018419 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] PATH_INFO: `/v3/users/f8616ecca9c543909db87c0ffcf511bc` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:59.019643 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8616ecca9c543909db87c0ffcf511bc)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:59.021635 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:59.066675 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8616ecca9c543909db87c0ffcf511bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.071926 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:05:59.074002 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8616ecca9c543909db87c0ffcf511bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.087660 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8616ecca9c543909db87c0ffcf511bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.088359 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8616ecca9c543909db87c0ffcf511bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.088966 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8616ecca9c543909db87c0ffcf511bc'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.101314 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1321/2613] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:05:58 2022] DELETE /identity/v3/users/f8616ecca9c543909db87c0ffcf511bc => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:05:59.279934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:05:59.281443 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:05:59.294575 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec5e30-a9b5-4d1b-8bf2-a3a3d6fe85a7 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:05:59.352263 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:05:59.352966 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:05:59.353146 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:05:59.353419 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] PATH_INFO: `/v3/projects/5f511c1474ea4a0787b6c7dfd68c16c9` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:05:59.354525 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f511c1474ea4a0787b6c7dfd68c16c9)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:05:59.356493 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:05:59.401549 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f511c1474ea4a0787b6c7dfd68c16c9', 'request_id': 'req-32280545-7fe1-46f4-b16b-c1b673c9d9db'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.407486 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f511c1474ea4a0787b6c7dfd68c16c9', 'request_id': 'req-32280545-7fe1-46f4-b16b-c1b673c9d9db'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.413599 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f511c1474ea4a0787b6c7dfd68c16c9', 'request_id': 'req-32280545-7fe1-46f4-b16b-c1b673c9d9db'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.425631 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] The token cache is being invalidate because project 5f511c1474ea4a0787b6c7dfd68c16c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93079) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:05:59.425917 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.426557 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:05:59.428056 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1293/2614] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:05:59 2022] DELETE /identity/v3/projects/5f511c1474ea4a0787b6c7dfd68c16c9 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:06:06.953999 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:06:06.955398 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:06:07.019909 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fadd1f7-a955-49ed-b7cb-be131b8bb1aa admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:06:07.072907 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:06:07.073512 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:06:07.073793 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:06:07.074069 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:06:07.075841 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:06:07.139108 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:06:07.140578 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:06:07.150481 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1322/2615] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:06:06 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:06:09.091842 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:06:09.093242 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:06:09.106176 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32280545-7fe1-46f4-b16b-c1b673c9d9db admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:06:09.110033 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:06:09.110632 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:06:09.110981 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:06:09.111259 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:06:09.113145 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:06:09.123527 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:06:09.125260 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:06:09.134212 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1294/2616] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:06:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:06:17.146096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=79,keystone:DELETE=3,keystone:INSERT=1 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:06:19.129803 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=61,keystone:INSERT=2,keystone:DELETE=8 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:08:05.319107 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:08:05.320570 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:08:05.404070 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3108c712-aea6-4c8b-9216-18becf13611a service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:08:05.410617 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:08:05.411231 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:08:05.411525 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:08:05.411790 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:08:05.414381 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:08:05.443554 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:08:05.446595 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:08:05.461846 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1323/2617] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:08:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:08:08.056856 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:08:08.058940 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:08:08.076236 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5022ca-0406-465e-9e87-a05f34a55438 service neutron] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:08:08.082699 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4244858dfb464d029b6f6bed0491557d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:08:08.083422 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:08:08.083704 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:08:08.083976 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:08:08.085867 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:08:08.099516 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:08:08.100777 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:08:08.112409 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1295/2618] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:08:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:08:11.133082 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:08:11.143379 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5606ec2-7dd6-4c27-ba80-f89ab7df92f0 service nova] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:08:11.147310 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bfae94e0fa041c9802729370a810245', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:08:11.147837 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] REQUEST_METHOD: `GET` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:08:11.148115 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:08:11.148420 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:08:11.150103 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:08:11.212033 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:08:11.213495 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:08:11.224272 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1324/2619] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:08:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:08:17.336223 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:08:17.337741 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:08:17.352832 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a06290-2cf5-43bd-a81b-d944c0aa8320 service nova] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:08:17.358381 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:08:17.359097 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:08:17.359461 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:08:17.359761 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] PATH_INFO: `/v3/users/3cddd27c37514e74a38430c46cef16fa` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:08:17.361017 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3cddd27c37514e74a38430c46cef16fa)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:08:17.362963 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:08:17.409929 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3cddd27c37514e74a38430c46cef16fa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.416934 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3cddd27c37514e74a38430c46cef16fa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.429335 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3cddd27c37514e74a38430c46cef16fa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.430000 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3cddd27c37514e74a38430c46cef16fa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.430608 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3cddd27c37514e74a38430c46cef16fa'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.443639 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1296/2620] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:08:17 2022] DELETE /identity/v3/users/3cddd27c37514e74a38430c46cef16fa => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:08:17.613666 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:08:17.625153 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cea24b-b904-44fd-b833-f9921e5d3072 service neutron] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:08:17.687887 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:08:17.688548 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:08:17.688851 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:08:17.689139 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] PATH_INFO: `/v3/projects/30742e7b914243df8dd120079560a3de` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:08:17.690232 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] RBAC: Authorizing `identity:delete_project(project_id=30742e7b914243df8dd120079560a3de)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:08:17.692187 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:08:17.714287 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:SELECT=100 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:08:17.739032 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30742e7b914243df8dd120079560a3de', 'request_id': 'req-ba877d3d-11d3-4847-a399-e85762ef046f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.745218 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30742e7b914243df8dd120079560a3de', 'request_id': 'req-ba877d3d-11d3-4847-a399-e85762ef046f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.752975 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30742e7b914243df8dd120079560a3de', 'request_id': 'req-ba877d3d-11d3-4847-a399-e85762ef046f'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.763685 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] The token cache is being invalidate because project 30742e7b914243df8dd120079560a3de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:08:17.763971 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.764630 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.766096 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1325/2621] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:08:17 2022] DELETE /identity/v3/projects/30742e7b914243df8dd120079560a3de => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 24 09:08:17.770291 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:08:17.771732 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:08:17.785496 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6569c7-27e9-41a2-b474-bfae689bf738 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:08:17.841579 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:08:17.842119 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:08:17.842396 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:08:17.842667 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] PATH_INFO: `/v3/users/0d7d316d62334b55a73221a3cb488c31` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:08:17.843776 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d7d316d62334b55a73221a3cb488c31)` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:08:17.845715 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:08:17.892161 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d7d316d62334b55a73221a3cb488c31'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.899007 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d7d316d62334b55a73221a3cb488c31'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.904420 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:08:17.913757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d7d316d62334b55a73221a3cb488c31'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.914786 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d7d316d62334b55a73221a3cb488c31'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.915447 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.notifications [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d7d316d62334b55a73221a3cb488c31'} {{(pid=93079) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:17.930219 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1297/2622] 10.222.0.39 () {58 vars in 1224 bytes} [Mon Oct 24 09:08:17 2022] DELETE /identity/v3/users/0d7d316d62334b55a73221a3cb488c31 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:08:18.109810 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] Authenticating user token {{(pid=93080) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:08:18.111280 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.fernet_utils [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93080) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:08:18.124881 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba877d3d-11d3-4847-a399-e85762ef046f admin admin] Validating token access rules against request {{(pid=93080) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:08:18.181882 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0bb4ebf18f65413bbe4945aec1caead8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '03b1f5372fb94c0fb009656de2a99622', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93080) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:08:18.182504 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] REQUEST_METHOD: `DELETE` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:08:18.182780 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] SCRIPT_NAME: `/identity` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:08:18.183053 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] PATH_INFO: `/v3/projects/c7a6f9b4599c433fa5666799780fbadb` {{(pid=93080) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:08:18.185487 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] RBAC: Authorizing `identity:delete_project(project_id=c7a6f9b4599c433fa5666799780fbadb)` {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:08:18.187958 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] RBAC: Authorization granted {{(pid=93080) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:08:18.232667 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c7a6f9b4599c433fa5666799780fbadb', 'request_id': 'req-f7ace6c9-775c-4ae0-b807-43a468637bce'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:18.238817 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c7a6f9b4599c433fa5666799780fbadb', 'request_id': 'req-f7ace6c9-775c-4ae0-b807-43a468637bce'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:18.246405 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c7a6f9b4599c433fa5666799780fbadb', 'request_id': 'req-f7ace6c9-775c-4ae0-b807-43a468637bce'} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:18.257243 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] The token cache is being invalidate because project c7a6f9b4599c433fa5666799780fbadb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=93080) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 24 09:08:18.257537 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:18.258170 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG keystone.notifications [None req-f7ace6c9-775c-4ae0-b807-43a468637bce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=93080) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 24 09:08:18.259651 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: [pid: 93080|app: 0|req: 1326/2623] 10.222.0.39 () {58 vars in 1233 bytes} [Mon Oct 24 09:08:18 2022] DELETE /identity/v3/projects/c7a6f9b4599c433fa5666799780fbadb => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 24 09:08:27.926171 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:DELETE=3,keystone:SELECT=6,keystone:INSERT=1 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:08:28.254120 dv-cinder-master-3-iscsi devstack@keystone.service[93080]: DEBUG dbcounter [-] [93080] Writing DB stats keystone:DELETE=14,keystone:SELECT=66,keystone:INSERT=2 {{(pid=93080) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 24 09:10:09.272600 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] Authenticating user token {{(pid=93079) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 24 09:10:09.273995 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:10:09.412811 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7eac10f-54b9-47d5-aa89-1d73677a1aa8 admin admin] Validating token access rules against request {{(pid=93079) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 24 09:10:09.470857 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e05b2b-31b6-4be3-b5cf-241eef953111 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c28fe744b614210a21aca0a7395089f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1356ba882d344f3d9e3b5d0806720c8f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=93079) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 24 09:10:09.471457 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e05b2b-31b6-4be3-b5cf-241eef953111 service placement] REQUEST_METHOD: `GET` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 24 09:10:09.471754 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e05b2b-31b6-4be3-b5cf-241eef953111 service placement] SCRIPT_NAME: `/identity` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 24 09:10:09.472014 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e05b2b-31b6-4be3-b5cf-241eef953111 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=93079) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 24 09:10:09.473775 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.fernet_utils [None req-b2e05b2b-31b6-4be3-b5cf-241eef953111 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=93079) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 24 09:10:09.485679 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e05b2b-31b6-4be3-b5cf-241eef953111 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 24 09:10:09.487243 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e05b2b-31b6-4be3-b5cf-241eef953111 service placement] RBAC: Authorization granted {{(pid=93079) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 24 09:10:09.497294 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: [pid: 93079|app: 0|req: 1298/2624] 10.222.0.39 () {60 vars in 1365 bytes} [Mon Oct 24 09:10:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 24 09:10:19.492757 dv-cinder-master-3-iscsi devstack@keystone.service[93079]: DEBUG dbcounter [-] [93079] Writing DB stats keystone:SELECT=89 {{(pid=93079) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}